Welcome to the website for the ciQA (complex, interactive question answering) task at TREC! The task began as a part of the TREC 2006 question answering track, and will continue in TREC 2007. For information on the main task ("question series"), please visit NIST's website. The primary goal of the ciQA task is to promote the development of interactive systems capable of addressing complex information needs.
On "complex": Information needs in ciQA, called topics, will consist of a template, which provides the question in a canonical form, and a narrative, which elaborates on what the user is looking for, provides additional context, etc. In the template, items in brackets represent "slots" whose instantiation varies from topic to topic. For example:
Template: What evidence is there for transport of [drugs] from [Bonaire] to [the United States]?
Narrative: The analyst would like to know of efforts made to discourage narco traffickers from using Bonaire as a transit point for drugs to the United States. Specifically, the analyst would like to know of any efforts by local authorities as well as the international community.
On "interactive": Participants will have the opportunity to deploy a fully-functional Web-based QA system for evaluation. For each topic, a human assessor will spend five minutes interacting with each system.
Information
- ciQA 2007 Guidelines
- Join mailing list (same as main QA task)
Resources
- TREC 2006 ciQA topics
- TREC 2006 ciQA answer nuggets
- Nuggeteer: Marton and Radul's automatic evaluation method (HLT/NAACL 2006)
- POURPRE: Lin and Demner-Fushman's automatic evaluation method (HLT/EMNLP 2005)