Semantic Self-Consistency: Enhancing Language Model Reasoning via Semantic Weighting

TL;DR

This paper introduces semantic self-consistency, a method that leverages reasoning paths in language models to improve reasoning accuracy and robustness in complex tasks.

Contribution

It extends the self-consistency framework by incorporating semantic reasoning paths, leading to enhanced reasoning reliability and performance.

Findings

Improved accuracy on reasoning benchmarks

More robust reasoning paths identified

Enhanced model performance on complex tasks

Abstract

While large language models (LLMs) have rapidly improved their performance on a broad number of tasks, they still often fall short on reasoning tasks. As LLMs become more integrated in diverse real-world tasks, advancing their reasoning capabilities is crucial to their effectiveness in nuanced, complex problems. Wang et al.'s self-consistency framework reveals that sampling multiple rationales before taking a majority vote reliably improves model performance across various closed-answer reasoning tasks. Standard methods based on this framework aggregate the final decisions of these rationales but fail to utilize the semantic information detailed in the step-by-step reasoning paths. Our work introduces semantic self-consistency, enhancing this approach by incorporating and analyzing both the reasoning paths of these rationales in addition to their final decisions before taking a majority vote. These methods not only improve the reliability of reasoning paths but also cause more robust performance on complex reasoning tasks.