Large Language Models for Data Annotation and Synthesis: A Survey

TL;DR

This survey explores how large language models like GPT-4 can automate data annotation and synthesis, highlighting their utility, assessment methods, and challenges to advance machine learning data preparation.

Contribution

It provides a comprehensive taxonomy, review of learning strategies, and discusses challenges of using LLMs specifically for data annotation and synthesis.

Findings

LLMs can generate high-quality annotations across various data types.

Assessment methods for LLM-generated annotations are evolving.

Using LLMs reduces annotation costs and accelerates data preparation.

Abstract

Data annotation and synthesis generally refers to the labeling or generating of raw data with relevant information, which could be used for improving the efficacy of machine learning models. The process, however, is labor-intensive and costly. The emergence of advanced Large Language Models (LLMs), exemplified by GPT-4, presents an unprecedented opportunity to automate the complicated process of data annotation and synthesis. While existing surveys have extensively covered LLM architecture, training, and general applications, we uniquely focus on their specific utility for data annotation. This survey contributes to three core aspects: LLM-Based Annotation Generation, LLM-Generated Annotations Assessment, and LLM-Generated Annotations Utilization. Furthermore, this survey includes an in-depth taxonomy of data types that LLMs can annotate, a comprehensive review of learning strategies for models utilizing LLM-generated annotations, and a detailed discussion of the primary challenges and limitations associated with using LLMs for data annotation and synthesis. Serving as a key guide, this survey aims to assist researchers and practitioners in exploring the potential of the latest LLMs for data annotation, thereby fostering future advancements in this critical field.