Larger Is Not Always Better: Leveraging Structured Code Diffs for Comment Inconsistency Detection

TL;DR

This paper introduces a structured, sequence-based approach for detecting comment inconsistency in code, leveraging code change activities to improve accuracy over existing LLM-based methods.

Contribution

It proposes a novel JIT CCI detection method using CodeT5+ that captures code change sequences, outperforming state-of-the-art models on benchmark datasets.

Findings

Outperforms recent models by up to 13.54% in F1-Score.

Achieves 4.18% to 10.94% improvement over fine-tuned LLMs.

Effectively captures code change activities for better inconsistency detection.

Abstract

Ensuring semantic consistency between source code and its accompanying comments is crucial for program comprehension, effective debugging, and long-term maintainability. Comment inconsistency arises when developers modify code but neglect to update the corresponding comments, potentially misleading future maintainers and introducing errors. Recent approaches to code-comment inconsistency (CCI) detection leverage Large Language Models (LLMs) and rely on capturing the semantic relationship between code changes and outdated comments. However, they often ignore the structural complexity of code evolution, including historical change activities, and introduce privacy and resource challenges. In this paper, we propose a Just-In-Time CCI detection approach built upon the CodeT5+ backbone. Our method decomposes code changes into ordered sequences of modification activities such as replacing, deleting, and adding to more effectively capture the correlation between these changes and the corresponding outdated comments. Extensive experiments conducted on publicly available benchmark datasets-JITDATA and CCIBENCH--demonstrate that our proposed approach outperforms recent state-of-the-art models by up to 13.54% in F1-Score and achieves an improvement ranging from 4.18% to 10.94% over fine-tuned LLMs including DeepSeek-Coder, CodeLlama and Qwen2.5-Coder.