Computer science doctoral student Mike Schaekermann, Dr. Joslin Goh and Schaekermann’s cosupervisors, Professors Kate Larson and Edith Law, have received a best paper award at CSCW 18, the 21stACM Conference on Computer Supported Cooperative Work and Social Computing.
Their paper, titled “Resolvable vs. irresolvable disagreement: A study on worker deliberation in crowd work,” explores a common issue when working with crowdsourced data — classifying the data accurately into categories.
L–R: Professor Kate Larson, Mike Schaekermann and Professor Edith Law deliberate in real time as they resolve a disagreement. (Dr. Joslin Goh was unavailable for the photo.)
A common assumption is that data can be classified into predefined categories that would be largely agreed upon by human annotators. In practice, however, classification often can be ambiguous. Annotators disagree for a host of reasons that stem from fuzzy definitions, missing context and imprecise questions to contradictory evidence and multiple interpretations arising from annotators having different levels of experience and expertise.
In their study, the researchers designed and implemented a workflow that allowed for real-time deliberation when classifying data. In particular, groups of crowdworkers could reexamine cases where they disagreed, either to resolve the disagreement or to declare them as irresolvable if not all members agreed on one and only one answer after multiple rounds of deliberation.
“Their study is the first that looks at whether disagreements among crowdworkers can be resolved in the context of both objective and subjective classification,” said Dan Brown, Director of the David R. Cheriton School of Computer Science. “I especially like their focus on deliberation during the categorization process. These discussions, when annotators disagree, increase the ultimate accuracy of categorization.”
The paper will be presented during the main CSCW 18 conference, which runs from November 5–7, 2018 in the New York City area.
To learn more about this research, please see Mike Schaekermann, Joslin Goh, Kate Larson and Edith Law. 2018. Resolvable vs. irresolvable disagreement: A study on worker deliberation in crowd work. Proc. ACM Hum-Comput Interact 2, CSCW, Article 154 (November 2018), 19 pages.