PhD Seminar • Natural Language Processing | Information Retrieval • The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models

Please note: This PhD seminar will take place online.

Ronak Pradeep, PhD candidate
David R. Cheriton School of Computer Science

Supervisor: Professor Jimmy Lin

Large Language Models have greatly improved information access systems, especially with retrieval-augmented generation (RAG). However, evaluating RAG systems remains a challenge. We address this by introducing AutoNuggetizer, an automatic evaluation framework that uses LLMs to generate and assign “nuggets”, factual information units, to system answers. Our experiments show strong agreement between automatic scores and human-based evaluations, especially when specific components like nugget assignment are automated. These results pave the path for more reliable and scalable RAG evaluation and beyond.

Attend this PhD seminar virtually on Zoom.