Please note: This master’s thesis presentation will take place in EC1 1237.
Anna Zhelizniak, Master’s candidate
David R. Cheriton School of Computer Science
Supervisor: Professor James R. Wallace
Qualitative analysis depends on the researcher’s interpretation of the data. Large language models (LLMs) are increasingly used to support this work, but most tools ask the model to produce codes, themes or summaries that the researcher then reviews, so the model’s reading of the data can become the starting point of the analysis.
This thesis explores a different role for the model: evaluating interpretations the researcher has written. It presents Quarrant, a Google Docs add-on built on Iterative Thematic Inquiry, a method in which researchers write tentative interpretations early and revise them as they test them against the data. Quarrant evaluates each interpretation against reference data the researcher selects, marks passages that appear supported or potentially misleading, and explains each evaluation with quotations checked against the sources, either as the researcher writes or when the researcher asks about a selected passage. Five qualitative researchers used Quarrant over several days in their own projects, and each participant was analyzed as a separate case. Participants used the evaluations as prompts to re-examine their interpretations rather than as answers to adopt. Their trust depended on whether they could see the evidence and data searched, and their preference for automated or manual analysis changed with the stage of their work. Based on these findings, the thesis discusses design implications for LLM tools that evaluate researchers’ interpretations.