AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitions

TL;DR

AetherCode introduces a new, more challenging benchmark for evaluating LLMs' programming skills, addressing previous limitations by including difficult problems and high-quality test cases from top competitions.

Contribution

It presents AetherCode, a benchmark with harder problems and expert-validated test suites from premier programming contests, improving the assessment of LLMs' coding abilities.

Findings

AetherCode offers a more rigorous evaluation framework.

LLMs show a larger gap to human experts on AetherCode.

Benchmark sets a new standard for future code reasoning research.

Abstract

Competitive programming has emerged as a critical benchmark for evaluating the reasoning and coding capabilities of Large Language Models (LLMs). Despite impressive progress on existing benchmarks, we argue that current evaluations overstate model proficiency, masking a substantial gap between LLMs and elite human programmers. This gap arises from two key limitations: insufficient difficulty and scope of benchmark problems, and evaluation bias from low-quality test cases. To address these shortcomings, we present AetherCode, a new benchmark that draws problems from premier programming competitions such as IOI and ICPC, offering broader coverage and higher difficulty. AetherCode further incorporates comprehensive, expert-validated test suites built through a hybrid of automated generation and human curation, ensuring rigorous and reliable assessment. By combining challenging problem design with robust evaluation, AetherCode provides a more faithful measure of LLM capabilities and sets a new standard for future research in code reasoning.