SAPO: Self-Adaptive Process Optimization Makes Small Reasoners Stronger

TL;DR

SAPO is a novel self-adaptive process optimization method that enhances small language models by efficiently reducing the reasoner-verifier gap, leading to improved performance in mathematics and coding tasks.

Contribution

The paper introduces SAPO, a new approach that actively minimizes the reasoner-verifier gap in small language models, overcoming inefficiencies of Monte Carlo supervision.

Findings

SAPO outperforms existing self-evolution methods on math and code tasks.

SAPO effectively reduces the reasoner-verifier gap.

New benchmarks demonstrate SAPO's impact on verifier performance.

Abstract

Existing self-evolution methods overlook the influence of fine-grained reasoning steps, which leads to the reasoner-verifier gap. The computational inefficiency of Monte Carlo (MC) process supervision further exacerbates the difficulty in mitigating the gap. Motivated by the Error-Related Negativity (ERN), which the reasoner can localize error following incorrect decisions, guiding rapid adjustments, we propose a Self-Adaptive Process Optimization (SAPO) method for self-improvement in Small Language Models (SLMs). SAPO adaptively and efficiently introduces process supervision signals by actively minimizing the reasoner-verifier gap rather than relying on inefficient MC estimations. Extensive experiments demonstrate that the proposed method outperforms most existing self-evolution methods on two challenging task types: mathematics and code. Additionally, to further investigate SAPO's impact on verifier performance, this work introduces two new benchmarks for process reward models in both mathematical and coding tasks.