Information for


Jesse HoeyResearch interests

Professor Hoey works in artificial intelligence, affective computing, and health informatics. He is primarily interested in developing computational models of human interactions with machines, and in using these models to build artificially intelligent applications in healthcare. This research involves four main aspects. First, he works on decision theoretic planning, particularly on Markov decision processes (MDPs), and their partially observable counterparts, POMDPs. He is interested in learning these models from data, and on solving them, particularly for large state, action, and observation spaces. Second, he works on sensor-based recognition of human behaviour (including gesture, facial expression and gait/body posture) from dynamic sensor streams (including video). He is interested in task-oriented sensor stream analysis (e.g. computer vision), in which the goal is to optimise over the action/policy space for an automated agent.

Third, he works on computational models of social and emotional behaviours of humans. This work integrates sociology, social psychology, and affective computing. Lastly, he uses sensors, decision theoretic models, and computational social science, to build assistive systems for persons with physical and cognitive disabilities. In particular, he is interested in systems that help a person with dementia during activities of daily living (ADL) using cameras and other sensors to inform decision processes with multiple and competing objectives. He is particularly interested in detecting and responding to emotional states of persons with these assistive technologies.  

Professor Hoey's research is funded in part by the Canadian Foundation for Innovation, the American Alzheimer's Association and the Natural Sciences and Engineering Research Council of Canada. He collaborates closely with researchers in Canada (Toronto, Guelph), the USA (Duke, Indiana), the UK (Newcastle, Dundee, Belfast, Heriot-Watt, Edinburgh), France (INRIA/Sophia-Antipolis), Germany (Bielefeld), and Mexico (Puebla, Cuernevaca, Ensenada). His recent projects include a Bayesian generalisation of Affect Control Theory, an emotionally aware tutoring system, and emotionally aware camera-based system to help persons with dementia during handwashing, a tangible (multi-touch) device for art therapists, a haptic robotic device for stroke rehabilitation, and socially aware ambient displays that help with communication and situation awareness for persons with dementia.

Degrees and awards

BSc (McGill), MSc, PhD (British Columbia)
Microsoft/AAAI Distinguished Contribution Award, IJCAI Workshop on Assisted Cognition (2009); Best Paper Award, International Conference on Computer Vision Systems (2007); Canesta Computer Vision Contest Grand Prize Winner (2006); Horace Watson Medal, McGill University Physics (1992)

Industrial and sabbatical experience

Professor Hoey spent four years in Scotland as an Assistant Professor in the School of Computing at the University of Dundee. He is also an adjunct scientist at the Toronto Rehabilitation Institute, where he works on assistive technology projects in collaboration with the Intelligent Assistive Technologies and Systems Lab (IATSL). In 2014-2015, he will be spending a sabbatical year with the STARS (Spatio-Temporal Activity Recognition Systems) Research Group at INRIA in France.

Representative publications

Michelle E. Karg, Ali-Akbar Samadani, Rob Gorbet, Kolja Kuhnlenz , Jesse Hoey and Dana Kulic Body Movements for Affective Expression: A Survey of Automatic Recognition and Generation. IEEE Transactions on Affective Computing, 2014

Christian Peters, Thomas Hermann, Sven Wachsmuth and Jesse Hoey Automatic Task Assistance for People with Cognitive Disabilities in Brushing Teeth - A User Study with the TEBRA System. ACM Transactions on Accessible Computing, 5, 4, March, 2014

George Zhu, Daniel Lizotte and Jesse Hoey Scalable Approximate Policies for Markov Decision Process Models of Hospital Elective Admissions. Artificial Intelligence in Medicine, 61, 2014

Marek Grzes, Jesse Hoey, Shehroz Khan, Alex Mihailidis, Stephen Czarnuch, Dan Jackson and Andrew Monk Relational Approach to Knowledge Engineering for POMDP-based Assistance Systems as a Translation of a Psychological Model. Journal of Approximate Reasoning, 55, 1 part 1, January, 2014

James Tung, Heather Snyder, Jesse Hoey, Alex Mihailidis, Maria Carrillo and Jesus Favela Everyday Patient-Care Technologies for Alzheimer's Disease. Pervasive Computing, 12, 4, Oct-Dec, 2013

Jesse Hoey, Tobias Schroeder and Areej Alhothali Bayesian Affect Control Theory. Proc.of the Humaine Association Conference on Affective Computing and Intelligent Interaction, Geneva, Switzerland, 2013

Marek Grzes, Pascal Poupart and Jesse Hoey Isomorph-free Branch and Bound Search for Finite State Controllers. Proceedings of International Joint Conference on Artificial Intelligence, Beijing, China, 2013

Jesse Hoey, Craig Boutilier, Pascal Poupart, Patrick Olivier, Andrew Monk and Alex Mihailidis People, sensors, decisions: Customizable and adaptive technologies for assistance in healthcare. ACM Transactions on Interactive Intelligent Systems, 2, 4, New York, NY, December, 2012

Liming Chen, Jesse Hoey, Chris D. Nugent, Diane Cook and Zhiwen Yu Sensor-Based Activity Recognition. IEEE Transactions on Systems, Man, and Cybernetics - Part C: Applications and Reviews, 42, 6, 2012

Jesse Hoey, Xiao Yang, Eduardo Quintana and Jesus Favela LaCasa: Location And Context-Aware Safety Assistant. Proc. International Conference on Pervasive Computing Technologies for Healthcare, San Diego, May, 2012

Hoey, Jesse and Poupart, Pascal and von Bertoldi, Axel and Craig, Tammy and Boutilier, Craig and Mihailidis, Alex. Automated Handwashing Assistance for Persons with Dementia Using video and a Partially Observable Markov Decision Process. Computer Vision and Image Understanding, 114(5):503-519, 2010.


University of Waterloo
Contact information: