Think Thrice Before You Act: Progressive Thought Refinement in Large Language Models

TL;DR

This paper introduces Progressive Thought Refinement (PTR), a novel framework enabling large language models to iteratively improve their responses, leading to better accuracy and quality across diverse tasks without task-specific fine-tuning.

Contribution

The paper proposes PTR, a two-phase training method that enhances LLMs' ability to self-refine responses, improving performance and response quality in open-ended scenarios.

Findings

Performance improved from 49.6% to 53.5% on average across ten tasks

Significant enhancement in response quality for open-ended tasks

Effective without task-specific fine-tuning

Abstract

Recent advancements in large language models (LLMs) have demonstrated that progressive refinement, rather than providing a single answer, results in more accurate and thoughtful outputs. However, existing methods often rely heavily on supervision signals to evaluate previous responses, making it difficult to assess output quality in more open-ended scenarios effectively. Additionally, these methods are typically designed for specific tasks, which limits their generalization to new domains. To address these limitations, we propose Progressive Thought Refinement (PTR), a framework that enables LLMs to refine their responses progressively. PTR operates in two phases: (1) Thought data construction stage: We propose a weak and strong model collaborative selection strategy to build a high-quality progressive refinement dataset to ensure logical consistency from thought to answers, and the answers are gradually refined in each round. (2) Thought-Mask Fine-Tuning Phase: We design a training structure to mask the "thought" and adjust loss weights to encourage LLMs to refine prior thought, teaching them to implicitly understand "how to improve" rather than "what is correct." Experimental results show that PTR significantly enhances LLM performance across ten diverse tasks (avg. from 49.6% to 53.5%) without task-specific fine-tuning. Notably, in more open-ended tasks, LLMs also demonstrate substantial improvements in the quality of responses beyond mere accuracy, suggesting that PTR truly teaches LLMs to self-improve over time.