Please note: This PhD defence will take place in M3 3001.
Aref Jafari, PhD candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Ali Ghodsi
Deep neural networks have achieved remarkable success across natural language processing and computer vision. However, the growing scale of these models has led to prohibitive computational and memory requirements, making their deployment on resource-limited devices increasingly challenging. Knowledge Distillation (KD) has emerged as a powerful model compression technique that enables a smaller student network to learn from a larger, more expressive teacher model. Despite its effectiveness, classical KD faces several fundamental challenges: (i) the capacity gap problem, where the student struggles to learn from an overly large teacher; (ii) the checkpoint search problem, where the best-trained teacher is not necessarily the optimal one for distillation; (iii) the noisy teacher problem, where unreliable teacher outputs can degrade the student’s learning; and (iv) the generative distribution-mismatch problem (of which exposure bias is one facet), where students trained on a fixed set of teacher-generated sequences behave very differently when generating their own outputs at inference time. This thesis proposes five novel methods that address these challenges across three complementary paradigms: logit distillation, data distillation, and divergence-aware distillation.
In the logit distillation paradigm, we first propose Annealing-KD, which tackles the capacity gap by applying a dynamic temperature function exclusively to the teacher’s logits, constructing an entropy-based curriculum that gradually exposes the student to increasingly sharp targets. We prove that the entropy of the teacher’s temperature-scaled distribution is strictly monotonically increasing with temperature, providing a principled mechanism for controlling the sharpness of the supervisory signal. Second, we propose Progressive Knowledge Distillation (Pro-KD), which addresses both the capacity gap and the checkpoint search problem by having the student follow the teacher’s entire training trajectory rather than distilling from a single, fully converged checkpoint. Combined with an adaptive temperature schedule, Pro-KD creates a doubly smooth curriculum that eliminates the need for costly checkpoint searches. Third, we propose Continuation-KD, which generalizes Annealing-KD by replacing its rigid two-stage structure with a single scheduled objective inspired by the continuation method from optimization theory. Continuation-KD’s mechanism is an annealed hinge-loss margin: once the student’s residual to the scaled teacher falls below the margin, the distillation term is exactly flat there and produces no gradient, so noise-level differences from the teacher are not transferred to the student. We give two exact identities for the annealed objective (one for the gradient, one for the loss Hessian) that isolate the teacher’s contribution in a single term scaled linearly by the annealing factor and make precise what the hinge removes. We are explicit that the step from this structure to improved optimization and generalization is not established by our experiments. All three methods consistently outperform the vanilla-KD and TAKD baselines on the GLUE benchmark, CIFAR-10, and CIFAR-100.
Moving beyond white-box distillation, we apply the data distillation paradigm to citation-grounded question answering. Using ChatGPT as a black-box teacher, we synthesize a training corpus of citation-enriched answers and fine-tune two compact student models: FIDCiter (3B parameters) and LLaMACiter (13B parameters). LLaMACiter exceeds the teacher on human-rated informativeness across all four evaluation sets and FIDCiter on two of the four, and FIDCiter matches or exceeds it on in-domain citation quality, while both use much shorter instruction templates; the teacher retains an edge on citation quality out of domain.
Finally, we propose Maximum Divergence Knowledge Distillation (MDKD) for autoregressive language model distillation. MDKD uses a rejection-sampling mechanism to preferentially select tokens where the student underestimates the teacher, constructing training sequences concentrated in regions where the student underweights teacher-supported tokens. We prove that the stochastic acceptance branch samples exactly from the positive teacher–student residual, with acceptance probability equal to their total variation distance. This active, divergence-aware approach targets the distributional-coverage aspect of the training–inference mismatch in sequence-level distillation; because its training prefixes are teacher-generated, direct on-policy mitigation of exposure bias is left to future work.
Collectively, the five proposed methods trace a progression from passive distillation with static objectives to active, adaptive paradigms that tailor the supervisory signal to the student’s current learning state. By addressing the capacity gap, checkpoint search, noisy teacher, and generative distribution-mismatch problems through complementary mechanisms, this thesis advances the design of efficient, high-performing models that bridge the gap between large-scale deep learning and real-world deployment constraints.