Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Sampling

TL;DR

This paper introduces Guided Self-Improvement (GSI), a Socratic-guided sampling method that enhances the efficiency of LLM self-improvement by focusing on challenging queries, reducing computational costs, and overcoming tail distribution issues.

Contribution

The paper proposes GSI, a novel Socratic-guided sampling strategy that improves the efficiency of LLM self-improvement on complex queries, addressing tail narrowing without high computational costs.

Findings

GSI improves performance on diverse mathematical tasks.

GSI reduces computational overhead compared to brute-force sampling.

GSI maintains effectiveness on held-out tasks.

Abstract

Self-improvement methods enable large language models (LLMs) to generate solutions themselves and iteratively train on filtered, high-quality rationales. This process proves effective and reduces the reliance on human supervision in LLMs' reasoning, but the performance soon plateaus. We delve into the process and find that models tend to over-sample on easy queries and under-sample on queries they have yet to master. As iterations proceed, this imbalance in sampling is exacerbated, leading to a long-tail distribution where solutions to difficult queries almost diminish. This phenomenon limits the performance gain of self-improving models. A straightforward solution is brute-force sampling to balance the distribution, which significantly raises computational costs. In this paper, we introduce Guided Self-Improvement (GSI), a strategy aimed at improving the efficiency of sampling challenging heavy-tailed data. It leverages Socratic-style guidance signals to help LLM reasoning with complex queries, reducing the exploration effort and minimizing computational overhead. Experiments on four models across diverse mathematical tasks show that GSI strikes a balance between performance and efficiency, while also being effective on held-out tasks.