Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning

TL;DR

Double-Checker is a framework that improves slow-thinking LLMs by enabling them to self-critique and refine their solutions iteratively, significantly boosting reasoning accuracy on benchmarks.

Contribution

The paper introduces a novel self-critical fine-tuning method that enhances reasoning in slow-thinking LLMs through explicit critique and iterative solution refinement.

Findings

Increases pass@1 on AIME from 4.4% to 18.2%.

Demonstrates significant improvement across multiple reasoning benchmarks.

Validates the effectiveness of self-critique in LLM reasoning.

Abstract

While slow-thinking large language models (LLMs) exhibit reflection-like reasoning, commonly referred to as the "aha moment:, their ability to generate informative critiques and refine prior solutions remains limited. In this paper, we introduce Double-Checker, a principled framework designed to enhance the reasoning capabilities of slow-thinking LLMs by fostering explicit self-critique and iterative refinement of their previous solutions. By fine-tuning on our curated 1,730 self-critical instances, Double-Checker empowers long-CoT LLMs to iteratively critique and refine their outputs during inference until they evaluate their solutions as correct under self-generated critiques. We validate the efficacy of Double-Checker across a comprehensive suite of reasoning benchmarks, demonstrating that iterative self-critique significantly enhances the reasoning capabilities of long-CoT LLMs. Notably, our Double-Checker increases the pass@1 performance on challenging AIME benchmarks from 4.4% to 18.2% compared to the original long-CoT LLMs. These results highlight a promising direction for developing more trustworthy and effective LLMs capable of structured self-critique. Our codes and data are available at https://github.com/XinXU-USTC/DoubleChecker