HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guarantees

TL;DR

HyPAC is a novel adaptive routing method that minimizes annotation costs across multiple sources, including LLMs and humans, while providing strong error guarantees without relying on data distribution assumptions.

Contribution

It introduces HyPAC, a cost-efficient, distribution-free method for adaptive annotation routing with PAC error guarantees, calibrated via importance sampling and confidence bounds.

Findings

Reduced annotation cost by 78.51% in experiments

Achieved PAC guarantees on annotation error

Effectively balances cost and accuracy across sources

Abstract

Data annotation often involves multiple sources with different cost-quality trade-offs, such as fast large language models (LLMs), slow reasoning models, and human experts. In this work, we study the problem of routing inputs to the most cost-efficient annotation source while controlling the labeling error on test instances. We propose \textbf{HyPAC}, a method that adaptively labels inputs to the most cost-efficient annotation source while providing distribution-free guarantees on annotation error. HyPAC calibrates two decision thresholds using importance sampling and upper confidence bounds, partitioning inputs into three regions based on uncertainty and routing each to the appropriate annotation source. We prove that HyPAC achieves the minimum expected cost with a probably approximately correct (PAC) guarantee on the annotation error, free of data distribution and pre-trained models. Experiments on common benchmarks demonstrate the effectiveness of our method, reducing the annotation cost by 78.51\% while tightly controlling the annotation error.