From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labeling

TL;DR

This paper introduces AutoAnnotator, a multi-model cooperative annotation framework that reduces costs and improves accuracy in data labeling by combining large and small language models with reinforcement learning.

Contribution

The paper proposes a novel multi-model cooperative annotation paradigm and an automatic framework that integrates LLMs and SLMs with reinforcement learning for efficient data annotation.

Findings

AutoAnnotator reduces annotation costs by 74.15%.

AutoAnnotator improves annotation accuracy by 6.21%.

Outperforms existing open-source/API LLMs in various settings.

Abstract

Although the annotation paradigm based on Large Language Models (LLMs) has made significant breakthroughs in recent years, its actual deployment still has two core bottlenecks: first, the cost of calling commercial APIs in large-scale annotation is very expensive; second, in scenarios that require fine-grained semantic understanding, such as sentiment classification and toxicity classification, the annotation accuracy of LLMs is even lower than that of Small Language Models (SLMs) dedicated to this field. To address these problems, we propose a new paradigm of multi-model cooperative annotation and design a fully automatic annotation framework AutoAnnotator based on this. Specifically, AutoAnnotator consists of two layers. The upper-level meta-controller layer uses the generation and reasoning capabilities of LLMs to select SLMs for annotation, automatically generate annotation code and verify difficult samples; the lower-level task-specialist layer consists of multiple SLMs that perform annotation through multi-model voting. In addition, we use the difficult samples obtained by the secondary review of the meta-controller layer as the reinforcement learning set and fine-tune the SLMs in stages through a continual learning strategy, thereby improving the generalization of SLMs. Extensive experiments show that AutoAnnotator outperforms existing open-source/API LLMs in zero-shot, one-shot, CoT, and majority voting settings. Notably, AutoAnnotator reduces the annotation cost by 74.15% compared to directly annotating with GPT-3.5-turbo, while still improving the accuracy by 6.21%. Project page: https://github.com/Zhaiyuan-Ji/AutoAnnotator.