Activity recognition in intelligent environments could play a key role for supporting people in their activities of daily life. Partially observable Markov decision process (POMDP) models have been used successfully, for example, to assist people with dementia when carrying out small multi-step tasks such as hand washing. POMDP models are a powerful, yet flexible framework for modeling assistance that can deal with uncertainty and utility in a theoretically well-justified manner. Unfortunately, POMDPs usually require a very labour intensive, manual setup procedure. The SNAP project aims to build a knowledge driven method for automatically generating POMDP activity recognition and situated prompting systems for a complex tasks. It starts with a psychologically justified description of the task and the particular environment in which it is to be carried out that can be generated from empirical data. This is then automatically combined with a specification of the available sensors and effectors to build a working prompting system.
SNAP System
The SNAP system is online and can be found here. Sign up and get started! Or, watch the video below to see how it works.Videos
This screencast shows how to create a SNAP instance using our database model.
The following video is an example of a SNAP instance (for tea making) in action. Part of the POMDP belief state (marginals) is shown, along with sensor readings. The video shows an archetypal scenario in which a volunteer was asked to prepare a cup of tea. The participant was instructed to wait for prompts before doing each required behavior, but to respond to all prompts except recognition prompts. Whilst we acknowledge that the design of appropriate prompts (in terms of both modality and content) is crucial to the development of an effective practical system for supporting people with dementia, in this case our prompts were chosen only to be recognizable by our participants (i.e. as recall, recognition or affordance reminders). This is filmed in the Ambient Kitchen at Culture Lab
Papers
- Marek Grzes, Jesse Hoey, Shehroz Khan, Alex Mihailidis, Stephen Czarnuch, Dan Jackson and Andrew Monk Relational Approach to Knowledge Engineering for POMDP-based Assistance Systems as a Translation of a Psychological Model. Journal of Approximate Reasoning, 55, 1 part 1, January, 2014 (bibtex)
- Jesse Hoey and Marek Grzes Distributed Control of Situated Assistance in Large Domains with Many Tasks. Proc. International Conference on Automated Planning and Scheduling, Freiberg, Germany, 2011 (bibtex)
- Jesse Hoey, Thomas Ploetz, Dan Jackson, Patrick Olivier, Andrew Monk and Cuong Pham Rapid Specification and Automated Generation of Prompting Systems to Assist People with Dementia. Pervasive and Mobile Computing, 7, 3, June, 2011 (bibtex)
- Marek Grzes, Jesse Hoey, Shehroz Khan, Alex Mihailidis, Stephen Czarnuch, Dan Jackson and Andrew Monk Relational Approach to Knowledge Engineering for POMDP-based Assistance Systems with Encoding of a Psychological Model. Proc. ICAPS workshop on Knowledge Engineering for Planning and Scheduling, Freiburg, Germany, 2011 (bibtex)
- Jesse Hoey, Thomas Ploetz, Dan Jackson, Andrew Monk, Cuong Pham and Patrick Olivier SNAP: SyNdetic Assistance Processes. NIPS 2010 Workshop on Machine Learning for Assistive Technologies, Whistler, BC, 2010 (bibtex)
- Jesse Hoey, Craig Boutilier, Pascal Poupart, Patrick Olivier, Andrew Monk and Alex Mihailidis People, sensors, decisions: Customizable and adaptive technologies for assistance in healthcare. ACM Transactions on Interactive Intelligent Systems, 2, 4, New York, NY, December, 2012 (bibtex) (short workshop version)
This work is in collaboration with Andrew Monk and Patrick Olivier.
American Alzheimer's Association through the Everyday Technology for Alzheimers Care (ETAC) program, and the EPSRC Social Inclusion in the Digital Economy (SIDE) Hub.