Adjudicator: Correcting Noisy Labels with a KG-Informed Council of LLM Agents

TL;DR

Adjudicator is a novel neuro-symbolic system that leverages a dynamic Knowledge Graph and a multi-agent LLM council to automatically identify and correct noisy labels in industrial datasets, significantly improving data quality.

Contribution

The paper introduces a KG-informed multi-agent LLM architecture for high-precision label correction, demonstrating superior performance over baselines in industrial data validation.

Findings

Achieves 0.99 F1-score on the AlleNoise benchmark subset.

Outperforms single-LLM and non-KG council baselines.

Uses KG for perfect identification of complex structural errors.

Abstract

The performance of production machine learning systems is fundamentally limited by the quality of their training data. In high-stakes industrial applications, noisy labels can degrade performance and erode user trust. This paper presents Adjudicator, a system that addresses the critical data mining challenge of automatically identifying and correcting label noise and has been validated for production deployment. Adjudicator models this as a neuro-symbolic task, first constructing a dynamic Knowledge Graph (KG) to unify item context. This KG then informs a "Council of Agents," a novel multi-agent Large Language Model architecture where specialized agents debate and vote on a label's validity. We validate our system on a 1,000-item balanced subset of the AlleNoise benchmark. Our KG-informed model achieves a 0.99 F1-score, significantly outperforming a single-LLM baseline (0.48 F1) and a non-KG council (0.59 F1). Our analysis reveals this is due to a Precision, achieved by a novel override logic that uses the KG to perfectly identify complex, structural errors (complete Recall) -- a class of errors that baselines fail to find. This result demonstrates a robust and explainable system for automated, high-precision data verification, serving as a vital proof-of-concept for generating golden datasets in strictly governed industrial environments.