CASCADE: Detecting Inconsistencies between Code and Documentation with Automatic Test Generation

TL;DR

CASCADE is a tool that detects code-documentation inconsistencies by generating and executing tests from documentation, significantly reducing false positives and uncovering previously unknown issues in open-source projects.

Contribution

This paper introduces CASCADE, a novel LLM-based approach that improves inconsistency detection accuracy by cross-verifying generated tests and code, with practical validation on multiple programming languages.

Findings

CASCADE identified 13 new inconsistencies in open-source repositories.

10 of these inconsistencies were fixed after detection.

CASCADE effectively reduces false positives compared to prior methods.

Abstract

Maintaining consistency between code and documentation is a crucial yet frequently overlooked aspect of software development. Even minor mismatches can confuse API users, introduce new bugs, and increase overall maintenance effort. This creates demand for automated solutions that can assist developers in identifying code-documentation inconsistencies. However, since automatic reports still require human confirmation, false positives carry serious consequences: wasting developer time and discouraging practical adoption. We introduce CASCADE (Consistency Analysis for Source Code And Documentation through Execution), a novel tool for detecting inconsistencies with a strong emphasis on reducing false positives. CASCADE leverages Large Language Models (LLMs) to generate unit tests directly from natural-language documentation. Since these tests are derived from the documentation, any failure during execution indicates a potential mismatch between the documented and actual behavior of the code. To minimize false positives, CASCADE also generates code from the documentation to cross-check the generated tests. By design, an inconsistency is reported only when two conditions are met: the existing code fails a test, while the code generated from the documentation passes the same test. We evaluated CASCADE on a novel dataset of 71 inconsistent and 814 consistent code-documentation pairs drawn from open-source Java projects. Further, we applied CASCADE to additional Java, C#, and Rust repositories, where we uncovered 13 previously unknown inconsistencies, of which 10 have subsequently been fixed, demonstrating both CASCADE's precision and its applicability to real-world codebases.