ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification

TL;DR

ReVISE introduces a novel framework enabling large language models to self-verify and correct their reasoning processes during inference, significantly enhancing their reasoning accuracy without relying on external verifiers.

Contribution

The paper presents ReVISE, a new self-verification framework for LLMs that uses curriculum learning and confidence-aware decoding to improve reasoning performance at test time.

Findings

ReVISE improves reasoning accuracy across multiple tasks.

Self-verification reduces errors in reasoning outputs.

Efficient training with preference learning enhances correction capabilities.

Abstract

Self-awareness, i.e., the ability to assess and correct one's own generation, is a fundamental aspect of human intelligence, making its replication in large language models (LLMs) an important yet challenging task. Previous works tackle this by employing extensive reinforcement learning or rather relying on large external verifiers. In this work, we propose Refine via Intrinsic Self-Verification (ReVISE), an efficient and effective framework that enables LLMs to self-correct their outputs through self-verification. The core idea of ReVISE is to enable LLMs to verify their reasoning processes and continually rethink reasoning trajectories based on its verification. We introduce a structured curriculum based upon online preference learning to implement this efficiently. Specifically, as ReVISE involves two challenging tasks (i.e., self-verification and reasoning correction), we tackle each task sequentially using curriculum learning, collecting both failed and successful reasoning paths to construct preference pairs for efficient training. During inference, our approach enjoys natural test-time scaling by integrating self-verification and correction capabilities, further enhanced by our proposed confidence-aware decoding mechanism. Our experiments on various reasoning tasks demonstrate that ReVISE achieves efficient self-correction and significantly improves reasoning performance.