Ranked Voting based Self-Consistency of Large Language Models

TL;DR

This paper introduces a ranked voting approach for large language models that generates and votes on multiple ranked answers to improve reasoning accuracy and self-consistency.

Contribution

It proposes a novel method of generating ranked answers and applying ranked voting techniques to enhance large language model reasoning performance.

Findings

Outperforms baseline methods on six datasets.

Using ranked voting improves reasoning accuracy.

Leverages multiple answers for more reliable self-consistency.

Abstract

Majority voting is considered an effective method to enhance chain-of-thought reasoning, as it selects the answer with the highest "self-consistency" among different reasoning paths (Wang et al., 2023). However, previous chain-of-thought reasoning methods typically generate only a single answer in each trial, thereby ignoring the possibility of other potential answers. As a result, these alternative answers are often overlooked in subsequent voting processes. In this work, we propose to generate ranked answers in each reasoning process and conduct ranked voting among multiple ranked answers from different responses, thereby making the overall self-consistency more reliable. Specifically, we use three ranked voting methods: Instant-runoff voting, Borda count voting, and mean reciprocal rank voting. We validate our methods on six datasets, including three multiple-choice and three open-ended question-answering tasks, using both advanced open-source and closed-source large language models. Extensive experimental results indicate that our proposed method outperforms the baselines, showcasing the potential of leveraging the information of ranked answers and using ranked voting to improve reasoning performance. The code is available at https://github.com/szu-tera/RankedVotingSC.