Not the Example, but the Process: How Self-Generated Examples Enhance LLM Reasoning

TL;DR

This paper investigates how self-generated examples improve reasoning in large language models, revealing that the act of creating problems, rather than the examples themselves, is the key factor behind performance gains.

Contribution

It introduces and empirically evaluates three prompting strategies, demonstrating that integrated prompting, which involves creating and solving problems within a single prompt, is most effective.

Findings

Integrated prompting outperforms zero-shot and decoupled prompting.

Self-generation benefits stem from problem creation, not the examples.

Attention patterns differ significantly between integrated and decoupled prompting.

Abstract

Recent studies have shown that Large Language Models (LLMs) can improve their reasoning performance through self-generated few-shot examples, achieving results comparable to manually curated in-context examples. However, the underlying mechanism behind these gains remains unclear, making it hard to decide when and how to apply the technique effectively. In this work, we argue that the key benefit arises not from the generated examples themselves but from the act of creating them. To validate this, on reasoning-intensive tasks across diverse LLM architectures, we systematically evaluate three prompting strategies for in-context learning: (1) Zero-shot prompting; (2) Integrated prompting, where LLMs create and solve problems within a single, unified prompt; and (3) Decoupled prompting, where self-generated examples are reused as in-context examples, but the context of their creation itself is excluded. We conduct experiments across five widely used model architectures, demonstrating that Integrated prompting consistently outperforms both Zero-shot and Decoupled prompting. In contrast, Decoupled prompting offers only marginal gains over Zero-shot. Further, for a more in-depth analysis, we conduct an attention analysis and observe significant differences in attention patterns between Integrated and Decoupled prompting. These findings suggest that the advantage of self-generation prompting comes from the process of problem creation, not the examples themselves, providing valuable insights for designing more effective prompting strategies.