Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming

TL;DR

This paper emphasizes the importance of separating problem-solving reasoning from code implementation in evaluating LLMs for competitive programming, proposing editorial-based evaluation to better diagnose reasoning capabilities.

Contribution

It introduces a novel editorial-based evaluation framework, a new dataset with gold editorials, and demonstrates the benefits of focusing on reasoning over code execution in benchmarking LLMs.

Findings

Generating editorials improves solve rates for some LLMs.

Expert editorials significantly boost model performance.

Models struggle with implementation even when given gold editorials.

Abstract

Large Language Models (LLMs) increasingly succeed on competitive programming problems, yet existing evaluations conflate algorithmic reasoning with code-level implementation. We argue that competitive programming is fundamentally a problem-solving task and propose centering natural-language editorials in both solution generation and evaluation. Generating an editorial prior to code improves solve rates for some LLMs, with substantially larger gains when using expertly written gold editorials. However, even with gold editorials, models continue to struggle with implementation, while the gap between generated and gold editorials reveals a persistent problem-solving bottleneck in specifying correct and complete algorithms. Beyond pass/fail metrics, we diagnose reasoning errors by comparing model-generated editorials to gold standards using expert annotations and validate an LLM-as-a-judge protocol for scalable evaluation. We introduce a dataset of 83 ICPC-style problems with gold editorials and full test suites, and evaluate 19 LLMs, arguing that future benchmarks should explicitly separate problem solving from implementation.