CCISolver: End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency

TL;DR

This paper introduces CCISolver, an end-to-end LLM-based framework for detecting and repairing method-level code-comment inconsistencies, supported by a high-quality dataset, achieving state-of-the-art results and practical efficiency.

Contribution

It presents a new high-quality dataset CCIBench and an innovative end-to-end LLM-based framework CCISolver for effective detection and repair of code-comment inconsistencies.

Findings

State-of-the-art detection F1-score of 89.54%

18.84% relative improvement in fixing GLEU score

36% faster inference speed than baseline

Abstract

Comments within code serve as a crucial foundation for software documentation, facilitating developers to communicate and understand the code effectively. However, code-comment inconsistency (CCI) can negatively affect software development, testing, and maintenance. Recent efforts to mitigate this issue have emerged, but existing studies often suffer from inaccurate datasets and inadequate solutions, weakening their practical effectiveness. In this study, we first conduct a quantitative analysis of existing datasets, revealing a substantial portion of sampled data are mislabeled. To address these data limitations, we introduce CCIBench, a refined dataset comprising high-quality data, to support the training and evaluation of method-level CCI methods. Furthermore, we present an innovative end-to-end LLM-based framework, CCISolver, designed to improve code quality by identifying and rectifying CCIs. Comprehensive evaluations demonstrate CCISolver's superior performance. For detection, it establishes a new state-of-the-art with an F1-score of 89.54%. In fixing task, it achieves a remarkable 18.84% relative improvement in GLEU score over the strongest baseline. This superiority is confirmed by human evaluation, where CCISolver's fixing success rate of 0.6533 significantly surpasses existing methods. Critically, in a practical end-to-end setting, CCISolver's innovative architecture is approximately 36% faster for inference than the baseline model, underscoring its scalability and real-world applicability.