Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling

TL;DR

RASC improves the efficiency and faithfulness of Large Language Model reasoning by dynamically evaluating and selecting optimal reasoning paths, reducing sample usage by 70% while maintaining accuracy.

Contribution

This paper introduces RASC, a novel framework that uses dynamic assessments to guide early stopping and rationale selection in LLM reasoning, enhancing efficiency and faithfulness.

Findings

Reduces sample usage by approximately 70%.

Maintains high accuracy with fewer samples.

Improves the faithfulness of LLM outputs.

Abstract

Self-Consistency mitigates hallucinations in Large Language Models (LLMs) by sampling multiple reasoning paths,but it lacks a systematic approach to determine the optimal number of samples or select the most faithful rationale. To address this limitation, we introduce Reasoning-Aware Self-Consistency (RASC), a novel framework that enhances sampling efficiency and reasoning faithfulness by dynamically evaluating both outputs and rationales. RASC assesses the quality of reasoning and the consistency of answers for each generated sample, using these assessments to guide early stopping decisions and rationale selection. The framework employs criteria-based stopping and weighted majority voting, enabling more informed choices on when to halt sampling and which rationale to select. Our comprehensive experiments across diverse question-answering datasets demonstrate that RASC outperforms existing methods, reducing sample usage by approximately 70% while maintaining accuracy. Moreover, RASC facilitates the selection of high-fidelity rationales, thereby improving the faithfulness of LLM outputs. Our approach effectively addresses the efficiency-accuracy trade-off in LLM reasoning tasks, offering a new perspective for more nuanced, faithful, and effective utilization of LLMs in resource-constrained environments.