MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks

TL;DR

MERA Code introduces a comprehensive benchmark for evaluating the practical coding skills of language models across multiple programming languages and tasks, addressing gaps in current evaluation methods.

Contribution

It presents a new multilingual, multi-task benchmark with an open-source framework, scoring system, and platform for standardized code generation evaluation.

Findings

Open LLMs show limitations in practical coding tasks.

Benchmark reveals language and task-specific performance gaps.

MERA Code facilitates standardized assessment and comparison.

Abstract

Advancements in LLMs have enhanced task automation in software engineering; however, current evaluations primarily focus on natural language tasks, overlooking code quality. Most benchmarks prioritize high-level reasoning over executable code and real-world performance, leaving gaps in understanding true capabilities and risks associated with these models in production. To address this issue, we propose MERA Code, a new addition to the MERA benchmark family, specifically focused on evaluating code for the latest code generation LLMs in Russian. This benchmark includes 11 evaluation tasks that span 8 programming languages. Our proposed evaluation methodology features a taxonomy that outlines the practical coding skills necessary for models to complete these tasks. The benchmark comprises an open-source codebase for users to conduct MERA assessments, a scoring system compatible with various programming environments, and a platform featuring a leaderboard and submission system. We evaluate open LLMs and frontier API models, analyzing their limitations in terms of practical coding tasks in non-English languages. We are publicly releasing MERA to guide future research, anticipate groundbreaking features in model development, and standardize evaluation procedures.