CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Tasks

TL;DR

CoreCodeBench is a novel, fine-grained, repository-level benchmark for evaluating code intelligence in LLMs, enabling detailed diagnosis of specific cognitive skills and addressing limitations of existing coarse-grained static benchmarks.

Contribution

It introduces CoreCodeBench and CorePipe, providing a configurable, high-quality, atomized task suite that isolates cognitive demands and supports difficulty scaling for better model evaluation.

Findings

LLMs show significant capability misalignment across cognitive dimensions.

Fine-grained evaluation reveals strengths and weaknesses not visible in coarse metrics.

CoreCodeBench achieves higher data validity and robustness compared to existing benchmarks.

Abstract

The evaluation of Large Language Models (LLMs) for software engineering has shifted towards complex, repository-level tasks. However, existing benchmarks predominantly rely on coarse-grained pass rates that treat programming proficiency as a monolithic capability, obscuring specific cognitive bottlenecks. Furthermore, the static nature of these benchmarks renders them vulnerable to data contamination and performance saturation. To address these limitations, we introduce CoreCodeBench, a configurable repository-level benchmark designed to dissect coding capabilities through atomized tasks. Leveraging our automated framework, CorePipe, we extract and transform Python repositories into a comprehensive suite of tasks that isolate distinct cognitive demands within identical code contexts. Unlike static evaluations, CoreCodeBench supports controllable difficulty scaling to prevent saturation and ensures superior data quality. It achieves a 78.55% validity yield, significantly surpassing the 31.7% retention rate of SWE-bench-Verified. Extensive experiments with state-of-the-art LLMs reveal a significant capability misalignment, evidenced by distinct ranking shifts across cognitive dimensions. This indicates that coding proficiency is non-monolithic, as strength in one aspect does not necessarily translate to others. These findings underscore the necessity of our fine-grained taxonomy in diagnosing model deficiencies and offer a sustainable, rigorous framework for evolving code intelligence. The code for CorePipe is available at https://github.com/AGI-Eval-Official/CoreCodeBench, and the data for CoreCodeBench can be accessed at https://huggingface.co/collections/tubehhh/corecodebench-68256d2faabf4b1610a08caa.