A research team led by Cheriton Professor Pascal Poupart and Carleton University’s Professor Sriram Ganapathi has received a $412,500 USD grant over two years from Coefficient Giving to make AI systems safer and stronger.
Coefficient Giving, formerly known as Open Philanthropy, is a philanthropic funder and advisor whose mission is to help others as much as possible. Since 2014, it has allocated more than $4 billion to support its focus areas, including global health and development, biosecurity, and AI safety and security.
What distinguishes Coefficient Giving is its unique research-based approach, which concentrates on problems that are important and tractable, but often neglected by other funders.
With this funding from Coefficient Giving, Professors Pascal Poupart (right) and Sriram Ganapathi (left) aim to make agentic systems safer and more reliable through constraint learning.
“While agentic systems open up many new exciting possibilities for workflow automation and productivity increases, they also introduce new safety risks. We are grateful for the visionary support of Coefficient Giving towards the exploration and prevention of those safety risks,” says Professor Poupart.
Professor Poupart, with project co-PI Professor Sriram Ganapathi and their PhD students and postdoctoral researcher, will focus on improving the reliability and safety of multi-agent orchestrators — systems that manage and coordinate multiple AI agents to perform complex tasks — in agentic systems
Although agentic systems are poised to accelerate workflows, there are concerns about their reliability and safety. Take an insurance company that wants to use AI to automate its underwriting process, customer service and claim approvals. Some possible consequences include misinformation from AI hallucinations or privacy breaches. There’s also uncertainty about whether the AI agents can perform the tasks effectively or comply with industry standards. Because of these issues, many companies are hesitant to deploy agentic systems.
The researchers are tackling this issue through constraint learning, a technique that implements rules, also known as constraints, for a system to follow. In this case, the constraints will detect and prevent unsafe actions, such as system failures, privacy leaks and harmful content. To train the AI agents, the team will create a dataset of safe and unsafe demonstrations. Ultimately, their techniques will be used to fine-tune the multi-agent orchestrator, leading to stronger agentic systems.
What sets their research apart is that they focus on unsafe behaviour that is unknown, implicit, or unidentifiable. For example, whether a user’s prompt could reveal harmful or private content. As a result, their approach could identify the root cause of unsafe behaviour. Additionally, past research has concentrated on safety issues during the deployment of large-language models (LLMs), whereas this work is centred on improving safety during LLM training.
With this funding from Coefficient Giving, Professor Poupart and his collaborators aim to make agentic systems safer and more reliable — driving an AI-integrated future.