Seed-CTS: Unleashing the Power of Tree Search for Superior Performance in Competitive Coding Tasks

TL;DR

This paper introduces Seed-CTS, a novel token-level tree search method that significantly improves competition-level code generation performance of large language models, especially when combined with Chain-of-Thought prompting.

Contribution

The paper presents a new tree search technique tailored for code generation, demonstrating substantial performance gains over existing models and methods.

Findings

Seed-CTS achieves pass@1 of 0.305 on LiveCodeBench-Hard.

Combining Chain-of-Thought prompting with Seed-CTS raises performance to 0.351.

Tree search significantly enhances competition-level code generation results.

Abstract

Competition-level code generation tasks pose significant challenges for current state-of-the-art large language models (LLMs). For example, on the LiveCodeBench-Hard dataset, models such as O1-Mini and O1-Preview achieve pass@1 rates of only 0.366 and 0.143, respectively. While tree search techniques have proven effective in domains like mathematics and general coding, their potential in competition-level code generation remains under-explored. In this work, we propose a novel token-level tree search method specifically designed for code generation. Leveraging Qwen2.5-Coder-32B-Instruct, our approach achieves a pass rate of 0.305 on LiveCodeBench-Hard, surpassing the pass@100 performance of GPT4o-0513 (0.245). Furthermore, by integrating Chain-of-Thought (CoT) prompting, we improve our method's performance to 0.351, approaching O1-Mini's pass@1 rate. To ensure reproducibility, we report the average number of generations required per problem by our tree search method on the test set. Our findings underscore the potential of tree search to significantly enhance performance on competition-level code generation tasks. This opens up new possibilities for large-scale synthesis of challenging code problems supervised fine-tuning (SFT) data, advancing competition-level code generation tasks.