The TREC 2006 ciQA task has concluded. This page is preserved purely for archival purposes. Click here for the most recent iteration of the ciQA task.
The ciQA (complex, interactive Question Answering) task is a secondary task in the TREC 2006 question answering track. For information on the main task ("question series"), please visit NIST's website.
The primary goal of the ciQA task is to promote the development of sytems capable of addressing complex information needs consisting of a template, which provides the question in a cannonical form, and a narrative, which elaborates on what the user is looking for, provides additional context, etc. For example:
Template: What evidence is there for transport of [drugs] from [Bonaire] to [the United States]?
Narrative: The analyst would like to know of efforts made to discourage narco traffickers from using Bonaire as a transit point for drugs to the United States. Specifically, the analyst would like to know of any efforts by local authorities as well as the international community.
The interaction component of the track will be optional. It is designed to provide an opporuntity to solict feedback from the human assessors through HTML forms that they interact with for three minutes. Both automatic and manual runs will be allowed
System responses will be evaluated using the "nugget pyramid" extension of the nugget-based methodology used in previous TRECs. The primary metric will be the F-score used in previous evaluations (see recent TREC overview papers). The track will also introduce a new secondary metric that attempts to quantify weighted nugget recall as a function of answer string length (see guidelines for more detail).
Information
- Guidelines
- Join mailing list (same as main QA task)
Resources
- Relationship questions from TREC 2005 in the 2006 ciQA task format.
- Answer key nuggets corresponding to the topics above.
- New ciQA training topics created by NIST for 2006.
- Nuggeteer: Marton's automatic evaluation method (HLT/NAACL 2006)
- POURPRE: Lin and Demner-Fushman's automatic evaluation method (HLT/EMNLP 2005)
Track Coordinators
- Diane Kelly
- Jimmy Lin