Please note: This master’s thesis presentation will take place online.
Vikram Subramanian, Master’s candidate
David R. Cheriton School of Computer Science
Supervisor: Professor Mei Nagappan
Bug localization is the process of identifying specific files in a codebase that contain defects based on bug reports. This thesis presents BugLLM, a novel zero-shot bug localization method leveraging Large Language Models (LLMs) and semantic search techniques. BugLLM comprises two main phases: ingestion and inference.
In the ingestion phase, the codebase is ingested and chunked using an Abstract Syntax Tree (AST) parser to preserve semantic continuity. These chunks are then embedded using OpenAI’s Ada V2 model and indexed in a Milvus vector database for efficient querying. In the inference phase, a query is generated from the bug description using an LLM with Chain-of-Thought prompting to filter out non-technical details. This refined query is used to search the vector database, retrieving the top 20 semantically similar code chunks. These chunks undergo further filtering using another LLM to establish their relevance to the bug, ensuring only the most pertinent chunks are considered.
Our method was evaluated on the Ye et al[60]. dataset, which includes bugs from six large Java projects. The evaluation metrics used include Top-5 accuracy, where BugLLM achieved a Top-5 accuracy ranging from 44.7% to 61.1%. BugLLM’s performance was competitive, often surpassing traditional methods like BugLocator, and demonstrated efficiency with no training required. To further aid developers, BugLLM also generates explanations for why specific files are relevant to a bug. The motivation behind this is twofold: helping developers under- stand why a file is important to fixing a bug and increasing transparency about how our tool works.
Our methodology employs Chain-of-Thought prompting to generate detailed explanations from LLMs. These explanations are evaluated based on technical accuracy, groundedness, and informativeness. We find that the explanations generated by BugLLM are largely accurate and grounded in the actual content and context of the code, with minimal hallucination. The explanations were also found to be informative, providing valuable insights to developers. The mean scores for technical accuracy, groundedness, and informativeness were 3.9, 4.5, and 4.2, respectively, across different prompting techniques.