CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks

TL;DR

This paper introduces CORE, a benchmark for evaluating large language models' ability to perform static analysis tasks in code, revealing strengths and weaknesses in semantic reasoning across multiple programming languages.

Contribution

The paper presents a new, human-verified benchmark with diverse static analysis tasks and a sampling strategy to evaluate LLMs' code reasoning capabilities across different languages.

Findings

LLMs excel at dependency identification but struggle with deep semantic understanding.

Models face challenges with complex control structures and backward dependencies.

The benchmark reveals specific areas for improving LLMs' reasoning in code analysis.

Abstract

Large language models (LLMs) have been widely adopted across diverse domains of software engineering, such as code generation, program repair, and vulnerability detection. These applications require understanding beyond surface-level code patterns: value propagation, control flow, and interdependence between program elements. However, existing benchmarks primarily evaluate end-to-end outcomes, such as whether code is correctly repaired or generated, leaving the models' ability for program semantic reasoning underexplored. This work presents CORE, a high-quality, human-verified benchmark designed to evaluate LLMs on fundamental static analysis tasks. CORE includes 12,553 task instances spanning data dependency, control dependency, and information flow across programs written in C/C++, Java, and Python. To ensure semantic diversity and reasoning complexity, we propose a semantics-aware diverse sampling strategy that selects targets and task instances based on structural coverage and dependency depth. We evaluate 10 mainstream LLMs and show that, while they perform well at identifying dependencies, models still struggle with tasks that require deeper semantic understanding and multi-step reasoning. We further conduct qualitative analyses to uncover key challenges, such as complex control structures and backward dependency patterns, offering insights into improving LLMs' code reasoning capabilities.