Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truth

TL;DR

This paper proposes a collaborative multi-model framework using diverse large language models to validate answers and assess question quality without relying on ground truth, leveraging inter-model agreement as a reliability indicator.

Contribution

It introduces a novel approach where multiple LLMs collaborate to produce and validate complex reasoning tasks without ground truth, using statistical measures to evaluate agreement and question quality.

Findings

Claude and Gemini produce more coherent questions with higher agreement.

LLAMA shows greater variability and lower consistency in question formulation.

Multi-model agreement correlates with answer reliability and question clarity.

Abstract

We introduce a new approach in which several advanced large language models-specifically GPT-4-0125-preview, Meta-LLAMA-3-70B-Instruct, Claude-3-Opus, and Gemini-1.5-Flash-collaborate to both produce and answer intricate, doctoral-level probability problems without relying on any single "correct" reference. Rather than depending on an established ground truth, our investigation focuses on how agreement among diverse models can signal the reliability of their outputs and, by extension, reflect the overall quality of the generated questions. To measure this inter-model alignment, we apply a suite of statistical evaluations, including chi-square tests, Fleiss' Kappa coefficients, and confidence interval calculations, thereby capturing both precision in answers and clarity in question phrasing. Our analysis reveals that Claude and Gemini tend to frame questions more coherently and unambiguously, which is evidenced by their tighter confidence intervals and greater concordance with responding agents. In contrast, LLAMA exhibits wider confidence bands and a lower level of agreement, indicating more variability and reduced consistency in its question formulations. These observations support the notion that a multi-model collaborative strategy not only improves answer dependability but also offers an effective, data-driven mechanism for evaluating and refining question quality when no definitive solution exists. Ultimately, this work delivers actionable insights into enhancing AI-guided reasoning processes through coordinated interactions among heterogeneous language models.