# Can Multi-turn Self-refined Single Agent LMs with Retrieval Solve Hard Coding Problems?

## TL;DR

This paper introduces the ICPC benchmark for competitive programming tasks and demonstrates that advanced inference techniques, including retrieval and self-judgment, significantly improve language models' problem-solving capabilities, even solving previously unsolvable problems.

## Contribution

The study develops a new benchmark and proposes novel inference methods that enhance language models' ability to solve complex programming problems.

## Key findings

- Zero-shot chain-of-thought achieves 19.1% pass@1
- Best technique reaches 42.2% pass@1
- Models can solve most previously unsolvable problems with specific instructions

## Abstract

Among the hardest tasks for humans are those found in competitive programming where problems require sophisticated algorithmic thinking, puzzle solving, and the creation of effective code. As a domain to assess language models (LMs), it has not received enough attention, though. This study presents the ICPC benchmark, which consists of 254 international collegiate programming contest (ICPC) tasks. Each problem includes official analysis, reference code, and sample, high-quality unit, and hidden tests. We are able to develop and evaluate a variety of LM inference techniques for competitive programming with these resources. With zero-shot chain-of-thought prompting, we find that o1 only achieves a 19.1\% pass@1 solve rate. With our best inference technique, which combines multi-turn self-judge with reflection and retrieval over episodic information, raises this to 42.2\%. Furthermore, we conduct a new human-in-the-loop investigation to gain a deeper understanding of the remaining difficulties. Surprisingly, we discover that o1 can solve 17 out of 18 problems that were previously unsolvable by any model or technique with just a few specific instructions. A footstep toward LMs with grounded, imaginative, and algorithmic thinking is provided by our quantitative findings and qualitative research. We open-source our code and data at https://github.com/kraritt/zolve.