Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs

TL;DR

This paper introduces a novel fine-tuning method for large language models that enables them to generate and refine multiple diverse reasoning chains within a single inference, leading to improved reasoning performance.

Contribution

The work presents a new approach called Diverse Chains of Thought (DCoT) that allows LLMs to perform in-inference refinement of reasoning chains, unlike prior methods.

Findings

Fine-tuning on DCoT improves performance across various tasks and model sizes.

Models can generate refined reasoning chains within a single inference step.

Significant gains observed in tasks with large result state spaces.

Abstract

Requiring a large language model (LLM) to generate intermediary reasoning steps, known as Chain of Thought (CoT), has been shown to be an effective way of boosting performance. Previous approaches have focused on generating multiple independent CoTs, combining them through ensembling or other post-hoc strategies to enhance reasoning. In this work, we introduce a novel approach where LLMs are fine-tuned to generate a sequence of Diverse Chains of Thought (DCoT) within a single inference step, which is fundamentally different from prior work that primarily operate on parallel CoT generations. DCoT allows LLMs to gain the ability to perform within-inference refinement of reasoning chains without requiring external feedback. Through a rigorous set of experiments spanning a wide range of tasks that require various reasoning types, we show that fine-tuning on DCoT improves performance over the CoT baseline across model families and scales (1.3B to 70B). These improvements are particularly impactful for tasks with a large result state space, such as those involving numeric answers. Our work is also significant because both quantitative analyses and manual evaluations reveal the observed gains stem from the models' ability to refine an initial reasoning chain by generating a second, improved chain within the same inference step, demonstrating previously elusive self-improvement. Our code and data are publicly available at https://github.com/UKPLab/acl2025-diverse-cot.