AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming

TL;DR

This paper introduces AlgoSimBench, a benchmark for testing LLMs' ability to identify algorithmically similar problems in competitive programming, revealing current limitations and proposing methods to improve detection accuracy.

Contribution

The paper presents AlgoSimBench, a new benchmark with annotated problems and a novel solution matching method to enhance algorithmic similarity detection in LLMs.

Findings

LLMs achieve only 65.9% accuracy on ASP identification.

Attempted solution matching improves accuracy by 6.7% to 11.7%.

Combining ASM with BM25 yields up to 52.2% accuracy.

Abstract

Recent progress in LLMs, such as reasoning models, has demonstrated strong abilities to solve complex competitive programming problems, often rivaling top human competitors. However, it remains underexplored whether these abilities generalize to relevant domains that are less seen during training. To address this, we introduce AlgoSimBench, a new benchmark designed to assess LLMs' ability to identify algorithmically similar problems (ASPs)-problems that can be solved using similar algorithmic approaches. AlgoSimBench consists of 1317 problems, annotated with 231 distinct fine-grained algorithm tags, from which we curate 402 multiple-choice questions (MCQs), where each question presents one algorithmically similar problem alongside three textually similar but algorithmically dissimilar distractors. Our evaluation reveals that LLMs struggle to identify ASPs, with the best-performing model (o3-mini) achieving only 65.9% accuracy on the MCQ task. To address this challenge, we propose attempted solution matching (ASM), a novel method for improving problem similarity detection. On our MCQ task, ASM yields an absolute accuracy improvement of 6.7% to 11.7% across different models. We also evaluated code embedding models and retrieval methods on similar problem identification. While the adversarial selection of problems degrades the performance to be less than random, we found that simply summarizing the problem to remove narrative elements eliminates the effect, and combining ASM with a keyword-prioritized method, BM25, can yield up to 52.2% accuracy. Code and data are available at github.com