Are Expert-Level Language Models Expert-Level Annotators?

TL;DR

This paper systematically evaluates large language models as expert-level data annotators across specialized domains, revealing their potential and limitations in expert tasks.

Contribution

It is the first comprehensive study assessing LLMs' performance as expert annotators in specialized fields, providing practical and cost-effective insights.

Findings

LLMs can perform at expert levels in certain specialized annotation tasks.

Performance varies significantly across different domains and tasks.

Practical guidelines for deploying LLMs as expert annotators are proposed.

Abstract

Data annotation refers to the labeling or tagging of textual data with relevant information. A large body of works have reported positive results on leveraging LLMs as an alternative to human annotators. However, existing studies focus on classic NLP tasks, and the extent to which LLMs as data annotators perform in domains requiring expert knowledge remains underexplored. In this work, we investigate comprehensive approaches across three highly specialized domains and discuss practical suggestions from a cost-effectiveness perspective. To the best of our knowledge, we present the first systematic evaluation of LLMs as expert-level data annotators.