CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs

TL;DR

CodeMMLU is a comprehensive benchmark with nearly 20,000 questions designed to evaluate code understanding and reasoning capabilities of Code Large Language Models across multiple tasks and programming languages.

Contribution

It introduces a new multi-task benchmark focused on assessing deep code comprehension and reasoning, addressing a gap in existing code evaluation methods.

Findings

State-of-the-art models perform poorly on CodeMMLU tasks.

CodeMMLU reveals significant gaps in code understanding beyond generation.

Benchmark covers diverse domains and programming languages.

Abstract

Recent advances in Code Large Language Models (CodeLLMs) have primarily focused on open-ended code generation, often overlooking the crucial aspect of code understanding and reasoning. To bridge this gap, we introduce CodeMMLU, a comprehensive multiple-choice benchmark designed to evaluate the depth of software and code comprehension in LLMs. CodeMMLU includes nearly 20,000 questions spanning diverse domains, including code analysis, defect detection, and software engineering principles across multiple programming languages. Unlike traditional benchmarks that emphasize code generation, CodeMMLU assesses a model's ability to reason about programs across a wide-range of tasks such as code repair, execution reasoning, and fill-in-the-blank challenges. Our extensive evaluation reveals that even state-of-the-art models struggle with CodeMMLU, highlighting significant gaps in comprehension beyond generation. By emphasizing the essential connection between code understanding and effective AI-assisted development, CodeMMLU provides a critical resource for advancing more reliable and capable coding assistants.