Write Your Own CodeChecker: An Automated Test-Driven Checker Development Approach with LLMs

TL;DR

AutoChecker is an LLM-powered system that automatically generates custom code checkers from rule descriptions and test suites, significantly improving effectiveness and applicability in real-world projects.

Contribution

The paper introduces AutoChecker, a novel LLM-based approach for automated, incremental, and logic-guided code checker generation from minimal input.

Findings

AutoChecker achieves an average test pass rate of 82.28%.

Generated checkers match the performance of official checkers in real-world projects.

AutoChecker outperforms five baseline methods across all effectiveness metrics.

Abstract

With the rising demand for code quality assurance, developers are not only utilizing existing static code checkers but also seeking custom checkers to satisfy their specific needs. Nowadays, various code-checking frameworks provide extensive checker customization interfaces to meet this need. However, both the abstract checking logic and the complex API usage of large-scale checker frameworks make this task challenging. To this end, automated code checker generation is anticipated to ease the burden of checker development. In this paper, we propose AutoChecker, an innovative LLM-powered approach that can write code checkers automatically based on only a rule description and a test suite. To achieve comprehensive checking logic, AutoChecker incrementally updates the checker's logic by focusing on solving one selected case each time. To obtain precise API knowledge, during each iteration, it leverages fine-grained logic-guided API-context retrieval, where it first decomposes the checking logic into a series of sub-operations and then retrieves checker-related API-contexts for each sub-operation. For evaluation, we apply AutoChecker, five baselines, and three ablation methods using multiple LLMs to generate checkers for 20 randomly selected PMD rules. Experimental results show that AutoChecker significantly outperforms others across all effectiveness metrics, with an average test pass rate of 82.28%. Additionally, the checkers generated by AutoChecker can be successfully applied to real-world projects, matching the performance of official checkers.