Verifying Large Language Models' Reasoning Paths via Correlation Matrix Rank

TL;DR

This paper introduces a novel internal indicator based on correlation matrix rank to verify LLM reasoning paths, reducing reliance on external resources and improving correctness detection with minimal computational overhead.

Contribution

It proposes a simple, effective correlation matrix rank-based method for internal verification of LLM reasoning paths, outperforming existing external resource-dependent approaches.

Findings

Achieves over 75% accuracy in correctness detection.

Improves reasoning benchmark accuracies by more than 8%.

Requires minimal computational overhead.

Abstract

Despite the strong reasoning ability of large language models~(LLMs), they are prone to errors and hallucinations. As a result, how to check their outputs effectively and efficiently has become a critical problem in their applications. Existing checking methods heavily rely on external resources, such as trained verifiers (e.g., process/outcome reward models) or elaborate prompts, which lead to high computational overhead and are only applicable to specific domains. In this paper, we investigate whether the internal behaviors of LLMs have already implied the credibility of their reasoning paths. Specifically, we find that the rank of the correlation matrix between the input problem and the output reasoning path is a robust indicator of reasoning correctness. Different from other correctness indicators for LLMs, the calculation of the correlation matrix only relies on the LLM itself, which avoids the hassle of training a separate model or designing complicated prompts. Based on it, we design a simple, plug-and-play Self-Indicator method to reweight candidate reasoning paths, which achieves significant performance improvements than other voting and verification methods with very few computational overhead. Our experiments across multiple LLMs of varying scales and model families have further shown the effectiveness of Self-Indicator. It achieves over 75% accuracy in distinguishing correct reasoning paths from incorrect ones, and, in turn, improves the accuracies on three reasoning benchmarks by more than 8%.