CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning

TL;DR

CodeSense introduces a comprehensive benchmark and dataset for fine-grained code semantic reasoning on real-world software projects, revealing current LLM limitations and providing tools for future research in software engineering tasks.

Contribution

It presents the first real-world code reasoning benchmark with a dataset from actual repositories, along with an execution tracing framework for future SE research.

Findings

State-of-the-art LLMs struggle with fine-grained code reasoning.

Prompting techniques improve but do not fully bridge the reasoning gap.

The dataset and tools facilitate future development of SE-focused models.

Abstract

Understanding and reasoning about code semantics is essential for enhancing code LLMs' abilities to solve real-world software engineering (SE) tasks. Although several code reasoning benchmarks exist, most rely on synthetic datasets or educational coding problems and focus on coarse-grained reasoning tasks such as input/output prediction, limiting their effectiveness in evaluating LLMs in practical SE contexts. To bridge this gap, we propose CodeSense, the first benchmark that makes available a spectrum of fine-grained code reasoning tasks concerned with the software engineering of real-world code. We collected Python, C and Java software projects from real-world repositories. We executed tests from these repositories, collected their execution traces, and constructed a ground truth dataset for fine-grained semantic reasoning tasks. We then performed comprehensive evaluations on state-of-the-art LLMs. Our results show a clear performance gap for the models to handle fine-grained reasoning tasks. Although prompting techniques such as chain-of-thought and in-context learning helped, the lack of code semantics in LLMs fundamentally limits models' capabilities of code reasoning. Besides dataset, benchmark and evaluation, our work produced an execution tracing framework and tool set that make it easy to collect ground truth for fine-grained SE reasoning tasks, offering a strong basis for future benchmark construction and model post training. Our code and data are located at https://codesense-bench.github.io/.