Asking the Right Questions: Improving Reasoning with Generated Stepping Stones

TL;DR

This paper explores how generating intermediate questions, called stepping stones, can improve large language models' reasoning abilities on complex tasks, by introducing a framework called ARQ and fine-tuning models for better question generation.

Contribution

It demonstrates the existence and transferability of effective stepping stone questions and proposes methods to fine-tune LLMs for generating more useful questions.

Findings

Good stepping stone questions exist and are transferable.

Generated questions significantly improve LLMs' problem-solving performance.

Fine-tuning enhances the quality of generated stepping stones.

Abstract

Recent years have witnessed tremendous progress in enabling LLMs to solve complex reasoning tasks such as math and coding. As we start to apply LLMs to harder tasks that they may not be able to solve in one shot, it is worth paying attention to their ability to construct intermediate stepping stones that prepare them to better solve the tasks. Examples of stepping stones include simplifications, alternative framings, or subproblems. We study properties and benefits of stepping stones in the context of modern reasoning LLMs via ARQ (Asking the Right Questions), a simple framework that introduces a question generator to the default reasoning pipeline. We first show that good stepping stone questions exist and are transferrable, meaning that good questions can be generated, and they substantially help LLMs of various capabilities in solving the target tasks. We next frame stepping stone generation as a post-training task and show that we can fine-tune LLMs to generate more useful stepping stones by SFT and RL on synthetic data.