The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotators

TL;DR

The ALCHEmist system uses program generation to produce labels from models, significantly reducing costs while maintaining or improving annotation quality compared to direct LLM-based labeling.

Contribution

We introduce a cost-effective method that generates reusable label-producing programs from models, outperforming traditional LLM annotation in accuracy and efficiency.

Findings

Achieves 12.9% performance improvement over LLM annotation.

Reduces total labeling costs by approximately 500 times.

Produces reusable, extendable label-generating programs.

Abstract

Large pretrained models can be used as annotators, helping replace or augment crowdworkers and enabling distilling generalist models into smaller specialist models. Unfortunately, this comes at a cost: employing top-of-the-line models often requires paying thousands of dollars for API calls, while the resulting datasets are static and challenging to audit. To address these challenges, we propose a simple alternative: rather than directly querying labels from pretrained models, we task models to generate programs that can produce labels. These programs can be stored and applied locally, re-used and extended, and cost orders of magnitude less. Our system, Alchemist, obtains comparable to or better performance than large language model-based annotation in a range of tasks for a fraction of the cost: on average, improvements amount to a 12.9% enhancement while the total labeling costs across all datasets are reduced by a factor of approximately 500x.