Hallucination Inspector: A Fact-Checking Judge for API Migration
Marcos Tileria, Santanu Kumar Dash, Profir-Petru P\^ar\c{t}achi, Earl T. Barr

TL;DR
This paper introduces Hallucination Inspector, a static analysis tool designed to detect hallucinated, incorrect code generated by LLMs during API migration tasks, improving accuracy over standard metrics.
Contribution
It presents a novel static analysis approach that verifies LLM-generated code against API documentation to identify hallucinations in software engineering.
Findings
Successfully detects hallucinated symbols in LLM-generated code
Reduces false positives compared to standard metrics
Effective in Android API migration scenarios
Abstract
Large Language Models (LLMs) are increasingly deployed in automated software engineering for tasks such as API migration. While LLMs are able to identify migration patterns, they often make mistakes and fail to produce correct glue code to invoke the new API in place of the old one. We call this issue Scaffolding Hallucination, a failure mode where models generate incorrect calling contexts by inventing Phantom Symbols -- such as imaginary imports, constructors, and constants -- that do not exist in the API specification. In this paper, we show that standard metrics cannot be relied upon to detect these instances of hallucination. We propose Hallucination Inspector, a static analysis tool to detect Scaffolding Hallucination in LLM-generated code. Our approach includes a lightweight evaluation framework that verifies symbols extracted from the abstract syntax tree against a knowledge…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
