Mirror-Consistency: Harnessing Inconsistency in Majority Voting

TL;DR

Mirror-Consistency improves Large Language Model reasoning and confidence calibration by enabling models to critically analyze minority responses and inconsistencies in ensemble decoding, surpassing traditional Self-Consistency methods.

Contribution

We introduce Mirror-Consistency, a novel method that enhances Self-Consistency by allowing models to reflect on inconsistencies, improving reasoning accuracy and confidence calibration.

Findings

Outperforms Self-Consistency in reasoning accuracy

Enhances confidence calibration in LLMs

Effectively identifies areas of uncertainty

Abstract

Self-Consistency, a widely-used decoding strategy, significantly boosts the reasoning capabilities of Large Language Models (LLMs). However, it depends on the plurality voting rule, which focuses on the most frequent answer while overlooking all other minority responses. These inconsistent minority views often illuminate areas of uncertainty within the model's generation process. To address this limitation, we present Mirror-Consistency, an enhancement of the standard Self-Consistency approach. Our method incorporates a 'reflective mirror' into the self-ensemble decoding process and enables LLMs to critically examine inconsistencies among multiple generations. Additionally, just as humans use the mirror to better understand themselves, we propose using Mirror-Consistency to enhance the sample-based confidence calibration methods, which helps to mitigate issues of overconfidence. Our experimental results demonstrate that Mirror-Consistency yields superior performance in both reasoning accuracy and confidence calibration compared to Self-Consistency.