Reflective Unit Test Generation for Precise Type Error Detection with Large Language Models

TL;DR

RTED is a novel type-aware test generation method that combines constraint analysis and reflection to detect Python type errors more accurately, reducing false positives and discovering new errors in real-world code.

Contribution

RTED introduces a new approach integrating type constraint analysis with reflective validation for precise Python type error detection.

Findings

Detects 22-29 more errors than existing techniques

Improves precision by up to 245.9%

Discovered 12 new errors in open-source projects

Abstract

Type errors in Python often lead to runtime failures, posing significant challenges to software reliability and developer productivity. Existing static analysis tools aim to detect such errors without execution but frequently suffer from high false positive rates. Recently, unit test generation techniques offer great promise in achieving high test coverage, but they often struggle to produce bug-revealing tests without tailored guidance. To address these limitations, we present RTED, a novel type-aware test generation technique for automatically detecting Python type errors. Specifically, RTED combines step-by-step type constraint analysis with reflective validation to guide the test generation process and effectively suppress false positives. We evaluated RTED on two widely-used benchmarks, BugsInPy and TypeBugs. Experimental results show that RTED can detect 22-29 more benchmarked type errors than four state-of-the-art techniques. RTED is also capable of producing fewer false positives, achieving an improvement of 173.9%-245.9% in precision. Furthermore, RTED successfully discovered 12 previously unknown type errors from six real-world open-source Python projects.