This is a screencast of the current graphical user interface:
This is a video of a prototype interface for the specification of the DIY Smart Home
From the user's side, our objective is to uncover and characterise/formalise the natural ways in which elderly persons living at home would be able to express the knowledge they have about their situation, preferences and needs. A natural method in this context is one that can uncover and make sense of the tacit knowledge people have about their home environment and how they interact with it. Many people have trouble describing things they do on a regular basis, or that which is encoded in procedural memory (e.g., describing in words how you drive your car), and we seek methods to uncover these descriptions in ways that are convenient, safe, efficient and ready-to-hand including, but not limited to:
- minimizing the need to overhaul physical artefacts in the home and how people interact with these and each other,
- seamlessly integrating into those current routines/habits of individual users and those involving/shared by multiple users, and
- designing affordances that support the above, or introducing new ways of doing things that are as intuitive to user(s) as possible.
However, we are not able yet to tackle this knowledge elicitation problem directly, as no methodology yet exists that is able to extract and characterise this information. The problem lies in the fact that we are trying to elicit knowledge from users about technology that does not exist yet, and that our end-users would not normally be able to conceptualise directly. Our research is therefore focussed on developing effective user-centered knowledge elicitation techniques in this space. As we are primarily looking at intelligent technologies to support older adults experiencing progressive cognitive decline or Alzheimer's dementia, our preliminary design approach involves co-designing with their family member, or "informal caregivers".
To help caregivers fully conceptualize the research and design problem, and start to grasp the potential of "smart" technologies, we argue for highly iterative user involvement, prototype development, and formative prototype evaluations. Presenting low-fidelity prototypes early and often throughout the design process can achieve:
- A clearer understanding of the research question(s), both for researchers/designers and users, where prototypes are utilized as probes to instantiate specific problems/scenarios and elicit the desired feedback/ideas/critiques
- Participatory design - proactive engagement, collaborative idea generation, and ownership by participants
- Future technology adoption and acceptance
From the technology side, our objective is to construct abstractions of intelligent smart-home sensing and control systems. These abstractions are formulated to represent domain knowledge in the most natural way possible for users. Details of the inner workings of the intelligent sensing and control systems are hidden from end users, who only need to provide information at a high enough level to be comfortable for them. Our research is focussed therefore on formalising these abstractions. Our formal models can eventually be used to build specific solutions for end users given the information extracted from our user-centered research stream.
It is important here to realise that the specific solutions referred to at the end of the last paragraph are not actual smart homes, but rather customised systems for end users to specify smart homes. Our user-centered research aims to uncover the language in which end users would be able to effectively specify their own situations and preferences, and our technology-centered research aims to provide a method to encode this language into an actual smart home system. In both streams of research, we are working a level which will allow this to generalise across multiple users. We are not developing the language for smart homes, we are developing formal models to uncover and encode the language for a specific user, and working towards making this possible for the widest range of users.
Overview
In D.I.Y. Smart Home, users provide information to the knowledge base about their needs and requirements as they arise. Our system will have a simple and intuitive interface, allowing them to quickly and easily describe a problem they are having (e.g. difficulty remembering to turn off the stove before leaving the kitchen) using the methods of IU analysis as in (see SNAP project or this paper). Their description will then be encoded as a set of fluents in the knowledge base. The knowledge base also links to technology developers and researchers by providing an interface for products in terms of their abilities and requirements in assistance tasks. Technologists provide information about their products and services by implementing this interface. This information is then also encoded in the knowledge base, and linked to the user's needs and requirements. %using the artificial intelligence techniques described above. The knowledge base therefore serves as an intelligent bridge between the users and technology developers, requiring each to only fill a particular interface describing their needs or abilities. A unique smart-home emerges for particular user through the slow process of the user's needs changing and being addressed by the knowledge base and technology developers.
Use Cases
Our intended users will be both older adults exhibiting early signs of cognitive impairment, and their family members or caregivers providing increasing support as a consequence. The proposed system will first be used by persons who are just entering the difficult transition into dementia, and their family members and caregivers. At this stage, a person may have some mild memory deficits, and concerned caregivers may be developing coping and support strategies while their loved one is still partially independent. This time period may arguably offer a unique opportunity to involve people in building their own smart home through the use of the intelligent knowledge base.This do-it-yourself approach is key to enabling smart-homes: instead of technologists building the homes and testing them once complete, we are allowing users themselves to slowly design and build their own smart homes, addressing problems as they arise during early disease onset.
Encoding of Knowledge
D.I.Y. Smart Home will encode the dynamics of assistance, user needs, sensor/technology capabilities in a logical knowledge base formulated as a probabilistic relational model (PRM). The PRM includes the goals, action preconditions, environment states, cognitive model, client and system actions (i.e., the outcome of the SNAP analysis), as well as relevant sensor models. A ground instance of the PRM is a POMDP, which we extract from a database using an automated procedure. A hierarchical approach can be used to connect the individual task-oriented POMDPs together into a single controller.
User Interface
D.I.Y. Smart Home will present a usable interface to end users (i.e., older adults dealing with memory issues and their concerned caregivers) allowing them to easily query the knowledge base with issues of relevance to their health needs, receiving information about products and services available to help with specific problems. The interface will also allow users to query medical information, engage in social networking, profit from collaborative filtering of assistive technologies and reviewing of products and services by others.
Knowledge Elicitation
We envision a video-game like interface where caregivers can perform an interactive care narrative, essentially telling a story about their care challenges, such as activities of daily living, date and time of day, location, affected stakeholders (i.e., family members), selected actions to to resolve problem, and feel- ings or ideas that arose from these problems. The idea of eliciting user preferences through narratives has been shown to a be a powerful method for eliciting detailed personal in- formation from users. See e.g. (Newell et al. 2006)
Developer Interface
D.I.Y. Smart Home will present an interface to technology providers (companies, researchers) to describe products according to abilities to fulfil assistance needs. The providers will populate the set of objects a usercan access by implementing an interface for each such object that defined its capabilities and requirements. Providers can also provide sensors or systems that can be retro-fit to existing devices, with which they are automatically paired. The providers will also be able to receive feedback on their products and services and search for required products and services not yet provided.Care Needs
D.I.Y. Smart Home will provide solutions to specific care needs as specified by end users by allowing them to directly purchase required products and services, and then linking these products and services into the care needs model developed for a specific person.
Cognitive Assistance
D.I.Y. Smart Home will, at run- time, provide assistance to a person by acquiring sensor data, querying the knowledge base, and providing appropriate as- sistance as specified by the POMDP controllers.
Economic Incentives
The knowledge base as a marketplace will bring the pressures of the free market to bear on the problem, forcing working solutions to be developed on an as-needed basis. The systems builds in a natural economic incentive, thus ensuring its uptake and adoption.
- Julie M Robillard, Wenkai Li, Shilpa Jacob, Dan Wang, Xin Zou and Jesse Hoey Co-Creating Emotionally Aligned Smart Homes Using Social Psychological Modeling. Proc. 4th international Workshop on Sensor-based Activity Recognition and Interaction (iWOAR), Rostock, Germany, 2017 (bibtex)
- Dan Wang and Jesse Hoey Hierarchical Task Recognition and Planning in Smart Homes with Partial Observability. Proc. 11th International Conference on Ubiquitous Computing and Ambient Intelligence , Philadelphia, PA, 2017 (bibtex)
- Amy Hwang, Michael Liu, Jesse Hoey and Alex Mihailidis DIY Smart Home: Narrowing the Gap Between Users and Technology. Proc. of the IUI Workshop on Interactive Machine Learning, Santa Monica, CA, 2013 (bibtex)
- Marek Grzes, Jesse Hoey, Shehroz Khan, Alex Mihailidis, Stephen Czarnuch, Dan Jackson and Andrew Monk Relational Approach to Knowledge Engineering for POMDP-based Assistance Systems as a Translation of a Psychological Model. Journal of Approximate Reasoning, 55, 1 part 1, January, 2014 (bibtex)
- Amy Hwang and Jesse Hoey Smart Home, The Next Generation: Closing the Gap between Users and Technology. AAAI Fall Symposium on AI for Gerontechnology, Washington, DC, 2012 (bibtex)
- Jesse Hoey and Marek Grzes Distributed Control of Situated Assistance in Large Domains with Many Tasks. Proc. International Conference on Automated Planning and Scheduling, Freiberg, Germany, 2011 (bibtex)
- Jesse Hoey, Thomas Ploetz, Dan Jackson, Patrick Olivier, Andrew Monk and Cuong Pham Rapid Specification and Automated Generation of Prompting Systems to Assist People with Dementia. Pervasive and Mobile Computing, 7, 3, June, 2011 (bibtex)
- Marek Grzes, Jesse Hoey, Shehroz Khan, Alex Mihailidis, Stephen Czarnuch, Dan Jackson and Andrew Monk Relational Approach to Knowledge Engineering for POMDP-based Assistance Systems with Encoding of a Psychological Model. Proc. ICAPS workshop on Knowledge Engineering for Planning and Scheduling, Freiburg, Germany, 2011 (bibtex)
- Jesse Hoey, Thomas Ploetz, Dan Jackson, Andrew Monk, Cuong Pham and Patrick Olivier SNAP: SyNdetic Assistance Processes. NIPS 2010 Workshop on Machine Learning for Assistive Technologies, Whistler, BC, 2010 (bibtex)