Patience Is The Key to Large Language Model Reasoning

TL;DR

This paper introduces a simple, test-time scaling method that encourages large language models to adopt a more patient and thorough reasoning style, improving complex problem-solving performance without extensive additional training.

Contribution

It proposes a preference optimization approach that promotes detailed reasoning in models through lightweight training, bridging the gap between brevity and complex reasoning capabilities.

Findings

Up to 2.1% performance increase on GSM8k

Effective in encouraging patient reasoning without extensive training

Uses lightweight dataset for training

Abstract

Recent advancements in the field of large language models, particularly through the Chain of Thought (CoT) approach, have demonstrated significant improvements in solving complex problems. However, existing models either tend to sacrifice detailed reasoning for brevity due to user preferences, or require extensive and expensive training data to learn complicated reasoning ability, limiting their potential in solving complex tasks. To bridge this gap, following the concept of scaling test-time, we propose a simple method by encouraging models to adopt a more patient reasoning style without the need of introducing new knowledge or skills. To employ a preference optimization approach, we generate detailed reasoning processes as positive examples and simple answers as negative examples, thereby training the model to favor thoroughness in its responses. Our results demonstrate a performance increase of up to 2.1% on GSM8k with training just on a lightweight dataset.