DocPrism: Local Categorization and External Filtering to Identify Relevant Code-Documentation Inconsistencies

TL;DR

DocPrism is a tool that detects code-documentation inconsistencies across multiple languages using LLMs, employing a novel filtering method to significantly reduce false positives and improve accuracy.

Contribution

It introduces the LCEF methodology to enhance LLM-based inconsistency detection, achieving high precision without fine-tuning across diverse programming languages.

Findings

LCEF reduces false positive rate from 98% to 14%.

Increases detection accuracy from 14% to 94%.

Maintains low flag rate of 15% with 0.62 precision across languages.

Abstract

Code-documentation inconsistencies are common and undesirable: they can lead to developer misunderstandings and software defects. This paper introduces DocPrism, a multi-language, code-documentation inconsistency detection tool. DocPrism uses a standard large language model (LLM) to analyze and explain inconsistencies. Plain use of LLMs for this task yield unacceptably high false positive rates: LLMs identify natural gaps between high-level documentation and detailed code implementations as inconsistencies. We introduce and apply the Local Categorization, External Filtering (LCEF) methodology to reduce false positives. LCEF relies on the LLM's local completion skills rather than its long-term reasoning skills. In our ablation study, LCEF reduces DocPrism's inconsistency flag rate from 98% to 14%, and increases accuracy from 14% to 94%. On a broad evaluation across Python, TypeScript, C++, and Java, DocPrism maintains a low flag rate of 15%, and achieves a precision of 0.62 without performing any fine-tuning.