Please note: This PhD seminar will take place online.
Qianqiu Zhang, PhD candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Ming Li
LLM-as-a-Judge is now widely used to check whether generated content is grounded in its sources. However, judges often accept unsupported claims with high confidence, and standard signals such as token probabilities, verbalized confidence and self-consistency cannot detect errors that are confident and consistent.
This talk reviews recent work on making these judgments more reliable. We first examine evidence that LLM judges are overconfident and produce high-certainty hallucinations. We then discuss statistical approaches, such as conformal prediction and false discovery rate control, which guarantee the error rate among accepted claims but require human-labeled calibration data for each new domain. Finally, we survey how synthetic negatives can reduce this labeling cost. We conclude by discussing what these methods mean for building LLM judges that are reliable and cheap to calibrate in practice.