Please note: This PhD seminar will take place in DC 3301.
Dake Zhang, PhD candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Mark D. Smucker
It is commonly believed that a free press is important to a society, but many people today lack trust in the news that they read. We envision that information retrieval can help people evaluate the trustworthiness of news by supporting the tested practice of lateral reading. Lateral reading is a process used by professional fact-checkers that involves asking questions about the sources of information and evidence present in a document and then searching for answers to these questions using search engines. Much as query completion or query suggestions can help people express their information needs, we see question generation as a first step towards helping people with the lateral reading process through automated systems. To be able to study question generation, we need to establish a suitable task and its evaluation.
In this paper, we report on our design of a question generation task as part of the TREC 2024 Lateral Reading Track and the evaluation of question-generation approaches for news trustworthiness assessment. Using data from this track, we evaluate the quality of questions from participating systems and compare them against questions generated by human experts. We find that while LLM-based systems can produce useful questions, human-generated questions are more diverse, and there is a gap between questions from humans and those from LLMs.