Please note: This master’s thesis presentation will take place in DC 2314 and online.
Ayinde Yakubu, Master’s candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Mei Nagappan
The evaluation uses historical development data from the Apache Kafka project obtained through the ApacheJIT dataset. From approximately 12,000 commit records, the dataset was filtered to obtain 524 one-to-one bug-inducing commit (BIC) and bug-fixing commit (BFC) relationships and 530 non-bug-inducing commits. An automated framework was developed to retrieve commit patches, submit code changes for LLM-based review, and record predictions and review comments. Three open-weight LLMs — gpt-oss-120b, gemma-4-31B-it, and Qwen3.6-35B-A3B — were evaluated under a common zero-shot prompting strategy across three repeated experimental runs. Performance was measured using precision, accuracy, recall, F1-score, balanced accuracy, Matthews correlation coefficient, and processing coverage. In addition, an LLM-as-a-Judge procedure assessed whether generated defect reports were semantically consistent with evidence from corresponding bug-fixing commits and Apache Kafka JIRA issue records.
The results show that the evaluated LLMs have limited reliability as autonomous defect detectors. Although the models identified subsets of historically labelled bug-inducing changes, substantial numbers of false positives and false negatives were observed. The first \texttt{gpt-oss-120b} run achieved the highest reported recall of 0.5163, while the highest individual-run accuracy was 0.4872. However, comparison with a trivial always-NOBUG baseline showed that model accuracies did not exceed the corresponding baseline accuracies on successfully processed records. Across the reported runs, balanced accuracy remained below 0.5 and Matthews correlation coefficient (MCC) remained negative, indicating weak overall discrimination between BIC and non-BIC benchmark examples.
The results further demonstrate that conventional classification metrics alone provide an incomplete characterisation of automated code-review quality, because useful review requires semantic correctness, actionable explanations, and sufficient project context. Semantic support was observed for approximately 16–23% of the evaluated first-run true-positive explanations. Thus, binary classification correctness does not necessarily imply that the generated explanation matches the historically documented defect.
The findings suggest that assistant-style use is a more appropriate direction for further evaluation than autonomous defect-oriented review. The thesis contributes a real-world evaluation framework, a comparative empirical assessment of three open-weight LLMs, and an evidence-based methodology for relating generated review comments to historically documented defects. The results also highlight repository context, semantic grounding, and hallucination reduction as important directions for improving future automated code review systems.
To attend this master’s thesis presentation in person, please go to DC 2314. You can also attend virtually on MS Teams.