Back to the Basics: Rethinking Issue-Commit Linking with LLM-Assisted Retrieval

TL;DR

This paper introduces a realistic evaluation framework for issue-commit linking, revealing the limitations of current deep learning methods, and proposes EasyLink, a new approach that significantly improves link retrieval accuracy using LLMs and vector databases.

Contribution

The paper presents a realistic dataset and evaluation setting for issue-commit linking, and introduces EasyLink, a novel method combining vector retrieval and LLM-based reranking to enhance accuracy.

Findings

Deep learning approaches' performance drops by over 50% in realistic settings.

Traditional VSM outperforms deep learning methods under realistic evaluation.

EasyLink achieves an average Precision@1 of 75.03%, surpassing previous methods by over four times.

Abstract

Issue-commit linking, which connects issues with commits that fix them, is crucial for software maintenance. Existing approaches have shown promise in automatically recovering these links. Evaluations of these techniques assess their ability to identify genuine links from plausible but false links. However, these evaluations overlook the fact that, in reality, when a repository has more commits, the presence of more plausible yet unrelated commits may interfere with the tool in differentiating the correct fix commits. To address this, we propose the Realistic Distribution Setting (RDS) and use it to construct a more realistic evaluation dataset that includes 20 open-source projects. By evaluating tools on this dataset, we observe that the performance of the state-of-the-art deep learning-based approach drops by more than half, while the traditional Information Retrieval method, VSM, outperforms it. Inspired by these observations, we propose EasyLink, which utilizes a vector database as a modern Information Retrieval technique. To address the long-standing problem of the semantic gap between issues and commits, EasyLink leverages a large language model to rerank the commits retrieved from the database. Under our evaluation, EasyLink achieves an average Precision@1 of 75.03\%, improving over the state-of-the-art by over four times. Additionally, this paper provides practical guidelines for advancing research in issue-commit link recovery.