Scattered Forest Search: Smarter Code Space Exploration with LLMs

TL;DR

Scattered Forest Search (SFS) is a novel code generation method that enhances exploration and feedback utilization, leading to higher success rates and efficiency compared to existing techniques.

Contribution

The paper introduces SFS, a new optimization-inspired search method for code generation that improves solution diversity and scalability over prior approaches.

Findings

Achieves 67.1% pass@1 on HumanEval+ with GPT-3.5, surpassing state-of-the-art by 8.6%.

Halves the number of iterations needed to find correct solutions.

Scales more efficiently than existing search methods.

Abstract

We frame code generation as a black-box optimization problem within the code space and demonstrate how optimization-inspired techniques can enhance inference scaling. Based on this perspective, we propose SCATTERED FOREST SEARCH (SFS), a novel approach that improves solution diversity and better exploits feedback during evolutionary search. Our theoretical analysis illustrates how these methods help avoid local optima during optimization, leading to more efficient exploration. Extensive experiments on HumanEval, MBPP, APPS, CodeContests, and Leetcode reveal significant performance gains. For instance, our method achieves a pass@1 rate of 67.1% on HumanEval+ and 87.2% on HumanEval with GPT-3.5, marking improvements of 8.6% and 4.3% over the state-of-the-art, while also halving the iterations needed to find the correct solution. Furthermore, our approach scales more efficiently than existing search techniques, including tree search, line search, and repeated sampling.