Atomic Self-Consistency for Better Long Form Generations

TL;DR

This paper introduces Atomic Self-Consistency (ASC), a novel method that enhances long-form language model responses by merging authentic subparts from multiple samples, significantly improving recall and factual accuracy over previous approaches.

Contribution

ASC is a new technique that improves long-form generation by merging relevant subparts from multiple stochastic samples, outperforming existing self-consistency methods.

Findings

ASC outperforms USC on multiple datasets

Merging subparts improves factual recall

Significant gains in open-ended QA tasks

Abstract

Recent work has aimed to improve LLM generations by filtering out hallucinations, thereby improving the precision of the information in responses. Correctness of a long-form response, however, also depends on the recall of multiple pieces of information relevant to the question. In this paper, we introduce Atomic Self-Consistency (ASC), a technique for improving the recall of relevant information in an LLM response. ASC follows recent work, Universal Self-Consistency (USC) in using multiple stochastic samples from an LLM to improve the long-form response. Unlike USC which only focuses on selecting the best single generation, ASC picks authentic subparts from the samples and merges them into a superior composite answer. Through extensive experiments and ablations, we show that merging relevant subparts of multiple samples performs significantly better than picking a single sample. ASC demonstrates significant gains over USC on multiple factoids and open-ended QA datasets - ASQA, QAMPARI, QUEST, ELI5 with ChatGPT and Llama2. Our analysis also reveals untapped potential for enhancing long-form generations using approach of merging multiple samples.