Please note: This PhD defence will be given online.
Alexandra Vtyurina, PhD candidate
David R. Cheriton School of Computer Science
Supervisors: Professors Charles Clarke and Edith Law
Natural language interfaces have seen a steady increase in their popularity over the past decade leading to the ubiquity of digital assistants. Such digital assistants include voice-activated personal assistants, such as Amazon’s Alexa, as well as text-based chat bots that can substitute for a human assistant in a call center. The main advantages of such systems are their ease of use and — in the case of voice-activated systems — hands-free interaction.
The majority of tasks undertaken by users of such commercially available systems are simple in nature, and the responses are often determined using a rules-based approach. However, such systems have the potential to support users in completing more complex and involved tasks. In this dissertation, we investigate user behaviours when interacting with natural language systems and study how improvements in design of such systems can benefit the user experience.
We begin by investigating social acceptance of, and user behaviour with, automated text-based conversational systems for information seeking tasks. To this end, in chapter three, we describe a study comparing three conversational agents used for information seeking tasks. In particular, we compare a human agent, an automated agent, and a perceived automated agent (implemented using a Wizard-of-Oz protocol). We evaluate the performance of each agent based on the participants’ preference. We also outline lessons learned and list design recommendations aimed to improve automated conversational search systems. We find no obvious biases against using an automated agent for information seeking tasks. In fact, our findings show that an automated system can be preferred to a conversation with a person in certain cases.
We continue the exploration of human behaviour with intelligent systems by examining another use case — cooking a culinary recipe. In chapter four, we examine the language users choose when interacting with a system with fairly limited capabilities. Supporting our findings of chapter three, we note that there is no apprehension in using an automated smart system. Additionally, we find that people use extremely rich and conversational language. We outline ways in which future systems can be designed to make use of this richness.
In chapter five, we shift our focus to the design of voice-based systems for web search. We investigate how individual search results can be presented using an audio-only communication channel, while taking into account significant differences between text and audio perception. We produce our audio representation based on a long established text representation. We subsequently compare user perception of audio and text result representations and outline ways in which audio representation of individual search results can be improved.
Finally, in chapter six, we demonstrate the versatility and usefulness of voice-based intelligent systems for populations with diverse abilities, in particular, for people who are visually impaired. We design a system for web search, called VERSE, that caters to people with visual impairments. Through a user survey, we demonstrate that people who are visually impaired are often heavy users and early adopters of voice-based technology and their insight and use cases can be beneficial to a wide range of the users of voice interfaces. We describe a design probe of an early stage prototype and discuss the benefits provided by the system, as well as possibilities for further improvement.
Overall, the research findings presented in this dissertation, discuss user perception of automated systems supporting non-trivial tasks and design recommendations for such systems.
To join this PhD thesis defence virtually on Zoom, please go to https://us02web.zoom.us/j/88963825536?pwd=SzZwaU5xTCtDeURhU1A5WjFnZ3ExUT09.
200 University Avenue West
Waterloo, ON N2L 3G1