Please note: This PhD defence will take place online.
Dongfu Jiang, PhD candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Wenhu Chen
Pretraining gives language models broad knowledge, but effective agents also need to learn how to use tools, respond to observations, and carry out tasks over many steps. This thesis examines three factors that shape these capabilities after pretraining: the signals used to evaluate and reward outputs, the trajectories used to demonstrate long-horizon behaviour, and the interfaces through which agents access external information. These factors constrain an agent in different ways. Verification determines which improvements a training objective can recognize; trajectories determine which behaviours are available to learn from; and interfaces determine what information and operations are available at run time. Their relative importance varies across settings, so improving one component may have little effect when another is the main source of error.
The first part presents methods we developed for constructing reliable evaluation signals without per-example human labels. These methods localize errors and combine parser-based checks with model judgements. For code, we used program execution to generate supervision, synthesizing 1.38 million test cases for 87.1 thousand programming problems. Training with these test cases improved a base model by 25 points on HumanEval-Plus in 80 optimization steps. Because a fixed test suite became less informative as the policy improved, we evolved test cases adversarially to maintain their discriminative power during training. For domains in which answers could not be executed, we extended the approach using a generative verifier trained on 230 thousand questions. Together, these studies expanded the coverage of automated verification while also showing its limits: fixed checks can become obsolete, and learned verifiers remain vulnerable to optimization.
Stronger verification alone was not sufficient for agentic reinforcement learning. When the initial policy rarely used tools effectively, its rollouts contained few successful behaviours for the learning algorithm to reinforce. The limiting factor in this setting was the availability of useful trajectories. To address it, we extracted 97 thousand long-horizon research trajectories from a capable teacher model in a reproducible offline search environment containing 15 million documents. Supervised fine-tuning on these trajectories, without reinforcement learning, improved a 30-billion-parameter model from 20.8% to 54.8% on BrowseComp-Plus.
In the final part, we held the model and corpus fixed and changed only the interface between them. Replacing top-k retrieval with direct, executable interaction over the corpus improved accuracy by 11 points while reducing cost by 29%. The improvement persisted when the retrieval baseline had already returned the relevant documents, indicating that the main benefit came from allowing the agent to inspect and manipulate evidence at a finer granularity rather than from retrieving different evidence. Taken together, these results show that progress in language agents depends not only on learning algorithms, but also on what can be verified, which behaviours are represented in training, and what the agent can access while it works.