This project aims to build an emotionally intelligent cognitive assistant to engage and help older adults with Alzheimer's disease (AD) to complete activities of daily living (ADL) more independently. Our new system combines two research streams. First, the development of cognitive assistants with artificially intelligent controllers using partially observable Markov decision processes (POMDPs, see the COACH project. Second, a model of the dynamics of emotion and identity call Affect Control Theory that arises from the sociological literature on culturally shared sentiments (see the BayesACT project. We have built a prototype assistive technology that combines the two. Work is in progress to understand further the emotional interaction between person's with Alzheimer's and automated prompting systems, and to refine the system and test it. Emotional or affective reasoning is one of the last major hurdles to overcome before intelligent cognitive assistants become widespread.

• Luyuan Lin
• Jesse Hoey
• Aarti Malhotra
• Celia Yu
• Stephen Czarnuch

• Alzheimer's Association
• AGEWELL Inc.
• Canadian Consortium on Neurodegeneration and Aging (CCNA)
• NSERC

These videos show an actor (in fact it is Luyuan Lin, the student who built the system) interacting with the emotionally aware handwashing system.
The system starts with the identities for client as shown (Elder with EPA of [1.67,0.01,-1.03] or Lonesome Elder with EPA of [[-0.64, -0.43, -1.81], significantly less "good", powerful and active than "elder"). The system has an identity of "assistant" (EPA [1.51,0.51,0.45]) except for in the last video where it is a "bossy assistant" (EPA [-0.89,0.52,0.53], significantly less good).
The system uses a Kinect camera and Stephen Czarnuch's hand tracker.

• Elder 1
• Elder 2
• Lonesome elder - showing different style of prompt at start only
• Loneseome Elder full sequence
• Bossy assistant with Elder

More details can be found by reading the following papers:

• Julie M Robillard and Jesse Hoey Emotion and Motivation in Cognitive Assistive Technologies for Dementia. Computer, 51, 3, March, 2018 (bibtex)
• Alexandra Konig, Linda E. Francis, Jyoti Joshi, Julie M Robillard and Jesse Hoey Qualitative study of affective identities in dementia patients for the design of cognitive assistive technologies. Journal of Rehabilitation and Assistive Technologies Engineering, 2017 (bibtex)
• Alexandra Konig, Aarti Malhotra, Jesse Hoey and Linda E. Francis Designing personalized prompts for a virtual assistant to support elderly care home residents. Proc. International Conference on Pervasive Computing Technologies for Healthcare, Cancun, Mexico, 2016 (bibtex)
• Alexandra Konig, Linda E. Francis, Aarti Malhotra and Jesse Hoey Defining affective Identities in elderly Nursing Home residents for the design of an emotionally intelligent cognitive assistant. Proc. International Conference on Pervasive Computing Technologies for Healthcare, Cancun, Mexico, 2016 (bibtex)
• Aarti Malhotra, Jesse Hoey, Alexandra Konig and Sarel van Vuuren A study of elderly people's emotional understanding of prompts given by Virtual Humans. Proc. International Conference on Pervasive Computing Technologies for Healthcare, Cancun, Mexico, 2016 (bibtex)
• Luyuan Lin, Stephen Czarnuch, Aarti Malhotra, Lifei Yu, Tobias Schroeder and Jesse Hoey Affectively Aligned Cognitive Assistance using Bayesian Affect Control Theory. Proc. of International Workconference on Ambient Assisted Living (IWAAL), Belfast, UK, 2014 (bibtex)
• Aarti Malhotra, Lifei Yu, Tobias Schroeder and Jesse Hoey An exploratory study into the use of an emotionally aware cognitive assistant. Proc. AAAI Workshop on artificial intelligence applied to assistive technologies and smart environments, Austin, TX, 2015 (bibtex)