LLMDFA: Analyzing Dataflow in Code with Large Language Models

TL;DR

LLMDFA introduces a novel, compilation-free dataflow analysis framework powered by large language models, enabling customizable and reliable analysis of code, including uncompilable programs, by decomposing tasks and leveraging external tools.

Contribution

This work presents LLMDFA, a new LLM-based framework that performs dataflow analysis without compilation, addressing hallucinations and improving applicability for real-world, evolving analysis needs.

Findings

Achieves 87.10% precision and 80.77% recall in bug detection

Surpasses existing techniques with up to 0.35 F1 score improvement

Effective on both synthetic and real-world Android programs

Abstract

Dataflow analysis is a fundamental code analysis technique that identifies dependencies between program values. Traditional approaches typically necessitate successful compilation and expert customization, hindering their applicability and usability for analyzing uncompilable programs with evolving analysis needs in real-world scenarios. This paper presents LLMDFA, an LLM-powered compilation-free and customizable dataflow analysis framework. To address hallucinations for reliable results, we decompose the problem into several subtasks and introduce a series of novel strategies. Specifically, we leverage LLMs to synthesize code that outsources delicate reasoning to external expert tools, such as using a parsing library to extract program values of interest and invoking an automated theorem prover to validate path feasibility. Additionally, we adopt a few-shot chain-of-thought prompting to summarize dataflow facts in individual functions, aligning the LLMs with the program semantics of small code snippets to mitigate hallucinations. We evaluate LLMDFA on synthetic programs to detect three representative types of bugs and on real-world Android applications for customized bug detection. On average, LLMDFA achieves 87.10% precision and 80.77% recall, surpassing existing techniques with F1 score improvements of up to 0.35. We have open-sourced LLMDFA at https://github.com/chengpeng-wang/LLMDFA.