M2G-Eval: Enhancing and Evaluating Multi-granularity Multilingual Code Generation

TL;DR

M2G-Eval is a comprehensive multilingual, multi-granularity benchmark for evaluating large language models' code generation capabilities across different code scopes and 18 programming languages, revealing insights into model performance and transferability.

Contribution

Introduces M2G-Eval, a novel multi-granularity, multilingual framework with new models and extensive evaluation to diagnose code generation abilities at various levels.

Findings

Line-level tasks are easiest for models.

Performance gaps widen with task complexity.

Models learn transferable programming concepts across languages.

Abstract

The rapid advancement of code large language models (LLMs) has sparked significant research interest in systematically evaluating their code generation capabilities, yet existing benchmarks predominantly assess models at a single structural granularity and focus on limited programming languages, obscuring fine-grained capability variations across different code scopes and multilingual scenarios. We introduce M2G-Eval, a multi-granularity, multilingual framework for evaluating code generation in large language models (LLMs) across four levels: Class, Function, Block, and Line. Spanning 18 programming languages, M2G-Eval includes 17K+ training tasks and 1,286 human-annotated, contamination-controlled test instances. We develop M2G-Eval-Coder models by training Qwen3-8B with supervised fine-tuning and Group Relative Policy Optimization. Evaluating 30 models (28 state-of-the-art LLMs plus our two M2G-Eval-Coder variants) reveals three main findings: (1) an apparent difficulty hierarchy, with Line-level tasks easiest and Class-level most challenging; (2) widening performance gaps between full- and partial-granularity languages as task complexity increases; and (3) strong cross-language correlations, suggesting that models learn transferable programming concepts. M2G-Eval enables fine-grained diagnosis of code generation capabilities and highlights persistent challenges in synthesizing complex, long-form code.