Please note: This PhD defence will take place online.
Kishanthan Thangarajah, PhD candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Mei Nagappan
AI-assisted coding tools powered by Code Large Language Models (CodeLLMs) have transformed software development, but their practical value depends on meeting stringent real-world constraints. Interactive tasks demand low latency, enterprises require self-hosted deployment to protect proprietary code, accelerator memory is limited and costly, and a model that performs well under one agent scaffold may degrade under another.
We investigate how to optimize the performance of CodeLLM-based coding assistants across these constraints, presenting four contributions. The Coding Assistant Task Orchestrator (CATO) is a service-level-agreement aware runtime that schedules requests and provisions accelerator replicas by each task’s latency profile, improving goodput and utilization while reducing tail latency. Context-Aware CodeLLM Eviction (CACE) is a model eviction policy that scores models on reload cost, future demand, and task criticality, reducing unnecessary data movement and cold starts in memory-constrained serving. Syntax-Aware Confidence Routing (SynConfRoute) is a training-free local-first routing method that pairs a small on-device CodeLLM with a larger model, combining token confidence with syntax validation to keep routine code completions private and local while escalating only difficult cases, exceeding the quality of the large model alone while cutting accelerator cost by more than half. Finally, Decoupling CLI Agent Scaffolding (DCAS) separates command-line-interface (CLI) agent scaffolds from backend models, showing that the cross-scaffold deployment gap is largely a planning-convention mismatch that can be internalized through fine-tuning to improve resolution rates and generalize to unseen scaffolds.
The first three optimize how the model is served, scheduling requests, keeping the right models resident in scarce accelerator memory, and escalating a request from a small local model to a larger one only when needed, while the fourth uses fine-tuning to make a model’s learned capability portable across scaffolds, together optimizing the performance of CodeLLM-based coding assistants under real-world deployment constraints.