DiversiGATE: A Comprehensive Framework for Reliable Large Language Models

TL;DR

DiversiGATE is a comprehensive framework for verifying large language models, integrating various methods and introducing a SelfLearner that improves accuracy through self-refinement, significantly enhancing performance on benchmarks like GSM8K.

Contribution

We propose DiversiGATE, a unified verification framework, and introduce SelfLearner, a novel model that learns from its outputs to improve LLM accuracy over time.

Findings

SelfLearner outperforms traditional LLMs on GSM8K.

Achieved a 54.8% to 61.8% accuracy improvement.

Framework consolidates multiple verification approaches.

Abstract

In this paper, we introduce DiversiGATE, a unified framework that consolidates diverse methodologies for LLM verification. The proposed framework comprises two main components: Diversification and Aggregation which provide a holistic perspective on existing verification approaches, such as Self-Consistency, Math Prompter and WebGPT. Furthermore, we propose a novel `SelfLearner' model that conforms to the DiversiGATE framework which can learn from its own outputs and refine its performance over time, leading to improved accuracy. To evaluate the effectiveness of SelfLearner, we conducted a rigorous series of experiments, including tests on synthetic data as well as on popular arithmetic reasoning benchmarks such as GSM8K. Our results demonstrate that our approach outperforms traditional LLMs, achieving a considerable 54.8% -> 61.8% improvement on the GSM8K benchmark.