Foundations of Global Consistency Checking with Noisy LLM Oracles

TL;DR

This paper introduces a scalable, polynomial-query complexity method for verifying global consistency of large fact collections using noisy LLM oracles, addressing the challenge of noise and combinatorial explosion.

Contribution

It formalizes the global consistency checking problem with noisy LLMs and proposes an adaptive divide-and-conquer algorithm to efficiently identify inconsistencies and minimal repairs.

Findings

Efficient detection and localization of inconsistencies in fact collections.

Low-degree polynomial query complexity of the proposed method.

Effective performance demonstrated on synthetic and real LLM oracles.

Abstract

Ensuring that collections of natural-language facts are globally consistent is essential for tasks such as fact-checking, summarization, and knowledge base construction. While Large Language Models (LLMs) can assess the consistency of small subsets of facts, their judgments are noisy, and pairwise checks are insufficient to guarantee global coherence. We formalize this problem and show that verifying global consistency requires exponentially many oracle queries in the worst case. To make the task practical, we propose an adaptive divide-and-conquer algorithm that identifies minimal inconsistent subsets (MUSes) of facts and optionally computes minimal repairs through hitting-sets. Our approach has low-degree polynomial query complexity. Experiments with both synthetic and real LLM oracles show that our method efficiently detects and localizes inconsistencies, offering a scalable framework for linguistic consistency verification with LLM-based evaluators.