Label Curation Using Agentic AI

TL;DR

AURA is an agentic AI framework that improves large-scale, multi-modal data annotation by jointly inferring true labels and annotator reliability without ground truth, significantly enhancing accuracy and reliability.

Contribution

It introduces AURA, a novel agentic AI system that automates and enhances data annotation by modeling annotator reliability and aggregating noisy labels without ground truth.

Findings

Achieves up to 5.8% accuracy improvement over baselines.

Up to 50% improvement in challenging scenarios with poor annotators.

Accurately estimates annotator reliability without pre-validation.

Abstract

Data annotation is essential for supervised learning, yet producing accurate, unbiased, and scalable labels remains challenging as datasets grow in size and modality. Traditional human-centric pipelines are costly, slow, and prone to annotator variability, motivating reliability-aware automated annotation. We present AURA (Agentic AI for Unified Reliability Modeling and Annotation Aggregation), an agentic AI framework for large-scale, multi-modal data annotation. AURA coordinates multiple AI agents to generate and validate labels without requiring ground truth. At its core, AURA adapts a classical probabilistic model that jointly infers latent true labels and annotator reliability via confusion matrices, using Expectation-Maximization to reconcile conflicting annotations and aggregate noisy predictions. Across the four benchmark datasets evaluated, AURA achieves accuracy improvements of up to 5.8% over baseline. In more challenging settings with poor quality annotators, the improvement is up to 50% over baseline. AURA also accurately estimates the reliability of annotators, allowing assessment of annotator quality even without any pre-validation steps.