The THEMIS.COG project is aimed at Theoretical and Empirical Modeling of Identity and Sentiments in Collaborative Groups. THEMIS.COG will provide new theoretical insights into the dynamics of self-organized collaborations, in which people come together to work on a common problem, without prompting by a third party. Understanding the social forces behind self-organized collaboration is increasingly important in today's society. Technological and social innovations are increasingly generated through informal, distributed processes of collaboration, rather than in formal, hierarchical organizations. Our work will use a data-driven approach to explore the social and psychological mechanisms that motivate self-organized collaborations and determine their likelihood of success or failure, focusing on the example of open, collaborative software development in online collaborative networks (OCNs) like GitHub.

The EMOTEC project, part of the AGEWELL NCE investigates and builds Emotional MOtivation for TEchnology that Cares. The objectives of EMOTEC are to develop and deploy computational models of emotional alignment between older adults with dementia, care partners, and technological assistants. Emotional alignment is a key element for the adoption and effectiveness of assistance, both human and non-human. The models we investigate are based in a social-psychological theory of the control of affect/emotion (called BayesACT) that has been extensively investigated in many different contexts, including in dementia and in technology. We propose three related sub-projects over the next two years as outlined below. Our objectives are to show that BayesACT is a solid basis for the modeling of alignment with technology in dementia more generally, and that this basis can be used to promote adoption and engagement with technology by older adults in contexts that span the AGEWELL remit..

The ACT@HOME project aims to build an emotionally intelligent cognitive assistant to engage and help older adults with Alzheimer's disease (AD) to complete activities of daily living (ADL) more independently. Our new system combines two research streams. First, the development of cognitive assistants with artificially intelligent controllers using partially observable Markov decision processes (POMDPs, see the COACH project.. Second, a model of the dynamics of emotion and identity call Affect Control Theory that arises from the sociological literature on culturally shared sentiments (see the BayesACT project. We have built a prototype assistive technology that combines the two. Work is in progress to understand further the emotional interaction between person's with Alzheimer's and automated prompting systems, and to refine the system and test it. Emotional or affective reasoning is one of the last major hurdles to overcome before intelligent cognitive assistants become widespread.

Affect Control Processes are a mathematical representation of the interactions between two persons, in which it is posited that people behave in a way so as to minimize the amount of deflection between their cultural emotional sentiments and the transient emotional sentiments that are created by each situation. It is a generalisation of Affect Control Theory, which presents a maximum likelihood solution in which optimal behaviours or identities can be predicted based on past interactions. In this project, we formulate a probabilistic and decision theoretic model of the same underlying principles, and show this to be a generalization of the basic theory. The new model, called BayesAct is more expressive than the original theory, as it can maintain multiple hypotheses about behaviours and identities simultaneously as a probability distribution, and can make value-directed action choices. This allows the model to generate affectively believable interactions with people by learning about their identity, predicting their behaviours, and taking actions that are simultaneously goal-directed and affect-sensitive.

This project is to design and build real-time systems to assist persons with dementia during handwashing. Assistance is given in the form of verbal and/or visual prompts, or through the enlistment of a human caregiver's help. The system uses only video inputs, and combines a Bayesian sequential estimation framework for tracking hands and towel, with a decision theoretic framework for computing policies of action -- specifically a partially observable Markov decision process (POMDP). A key element of the system is the ability to estimate and adapt to user states, such as awareness, responsiveness and overall dementia level.

This project aims to build a "do-it-yourself" version of a smart-home which connects users with developers by building a person-specific logical knowledge base of user needs, assistance dynamics, sensors, actuators and care solutions. The knowledge base will also serve as a run-time processor for the provision of assistance in specific tasks: it will act as the equivalent of a 'smart home', but will be a dynamically evolving variant, customisable in real-time by end users and product developers. Thus, in D.I.Y. Smart Home, the smart-home emerges from the specific requirements of a user in a do-it-yourself approach that gives control to the user, while allowing them to access technological solutions. Economic incentives are built directly into DIY Smart Home by proposing a marketplace for the proposal, construction and selling of smart-home technologies tailored to individual needs.

SNAP: SyNdetic Assistance Processes for Rapid Development of Technologies for Wellness. This project aims to develop a methodology for rapid prototyping of POMDP models for assistive technology. A POMDP for assistance of persons with dementia has system actions as the various prompts or memory aids the system can use to help a user remember things in the task. For example, shining a light on the kettle may remind them of the kettle's purpose. A key problem is the initial specification of the model for a particular task. We accomplish this by qualitative and quantitative task analysis using Syndetic modeling applied to a small number of example videos. The result is a proces for the initial specification of a full working assistive technology for a particular task. In collaboration with Andrew Monk and Patrick Olivier.

The LaCasa project, joint work with Jesus Favela, CICSE, Ensenada, Mexico, is to build activity recognition systems into smart phones for elderly persons with a cognitive disability. The activity recognition provides a stream of data on the mobile device that can be used to provide the user with situational awareness cues and to update their status on a protected social networking site (SNS), for the information of relevant stakeholders such as family members, or health professionals. Awareness information that can be obtained from sensors in the mobile device includes location, presence, or gestures. Family and friends can use the SNS to be aware of the elders whereabouts and decide when is a good moment to call, assist her remotely when at risk, add or modify entries to her schedule, or send affective messages to the mobile and/or ambient displays. Deciding what information to share with the different stakeholders is an important issue, and we propose a decision theoretic approach to determine what to information to share with whom. Our initial work on this topic is to build a Location And Context Aware Safety Assistant (LACASA).

This project is developing a real-time system that guides stroke patients during upper extremity rehabilitation. The system automatically modifies exercise parameters to account for the specific needs and abilities of different individuals. The system uses a partially observable Markov decision (POMDP) model of a rehabilitation exercise that can capture this form of customization.

SPUDD stands for Stochastic Planning Using Decision Diagrams. SPUDD implements a value iteration algorithm for MDPs and POMDPs that uses algebraic decision diagrams (ADDs) to represent value functions and policies.

This project aims to build novel tools that increase the capacity of arts therapists to engage cognitively disabled older people in artistic activities. Engagement with the arts is becoming widely accepted as a method for promoting quality of life. However, persons with Alzheimer's disease have difficulty engaging in a creative activity. Art therapists engage older adults in artistic activities that promote autonomy and independence, thereby possibly increasing the older adult's ability to engage with the world. Developed in conjunction with users, the tools are customizable touch-screen interface creative arts devices. A therapist's customized application can track a user's activities and can respond to the user using powerful artificial intelligence (AI) and computer vision algorithms.

This project, joint work with Johanna Moore, is to investigate integrating spoken dialogue into cognitive assistants, such as the COACH. Although these congnitive assistants provide verbal prompts, they cannot react to verbal user feedback. This is a major limitation, and we seek to enable these systems to understand feedback from a user and to respond appropriately. We are designing a POMDP-based assistance system for a calendar application using SNAP, and plan to deploy it in a wizard-of-Oz experiment to test it and to gather a corpus of data for building more effective spoken dialogue systems.

I work on the SIDE Hub, and specifically on Machine Learning for situated prompting systems in the home. This project uses the Ambient Kitchen at Culture Lab, Newcastle University, run by Patrick Olivier.

Multi-touch tables are becoming increasingly popular and much research is dedicated to developing suitable interaction paradigms. There also exist multiple techniques aimed at extending interactions into the hoverspace--the space directly above a multi-touch table. We propose a novel hoverspace method that does not require any additional hardware or modification of existing vision-based multi-touch tables. Our prototype system was developed on a Diffused Surface Illumination (DSI) vision-based multi-touch set up, and uses light reflected from a person's palm to estimate its position in 3D space above the table. Read more here.

The DyNaMo project investigates probabilistic graphical models (PGMs) and their applications. PGMs allow the representation of probability distributions with many variables in a compact form, also they help to make probabilistic inference (estimate the probability of certain variables given other known) efficiently. PGMs include: Bayesian classifiers, Bayesian networks, Markov random fields, and have many applications in medicine, expert systems, industrial diagnosis, image analysis, robotics and many others.

This project uses stereo vision to detect card counters in the game of Blackjack. The system tracks the game as it progresses, monitors the cards played, and tracks the player's betting patterns. The correlation between the player's betting patterns and the game card count is analysed to determine the likelihood that a player is card counting, and the system alerts the Casino staff upon positive identification of a card counter. The Blackjack tracking system also has the ability to detect dealer errors, by monitoring dealers' actions made during the game and ascertaining whether or not the correct action was taken.

This project aims to develop realistic large-scale SSDS with an accurate, extended representation of user goals, and to use new Automatic Belief Compression (ABC) techniques to plan over the large state spaces thus generated. Techniques such as Value-Directed Compression demonstrate that compressible structure can be found automatically in the SSDS domain. Joint work with Oliver Lemon and Paul Crook.