Towards Automating Text Annotation: A Case Study on Semantic Proximity Annotation using GPT-4

TL;DR

This study investigates automating semantic proximity annotation using GPT-4 with automatic prompting, emphasizing the importance of prompt design and demonstrating an open-source tool for practical annotation tasks.

Contribution

It introduces a method for designing automatic prompts based on human guidelines and implements these in an open-source annotation tool, advancing automated semantic annotation.

Findings

Prompt design significantly impacts annotation quality.

Small modifications to guidelines improve GPT-4 performance.

Automatic prompts can be effectively integrated into annotation workflows.

Abstract

This paper explores using GPT-3.5 and GPT-4 to automate the data annotation process with automatic prompting techniques. The main aim of this paper is to reuse human annotation guidelines along with some annotated data to design automatic prompts for LLMs, focusing on the semantic proximity annotation task. Automatic prompts are compared to customized prompts. We further implement the prompting strategies into an open-source text annotation tool, enabling easy online use via the OpenAI API. Our study reveals the crucial role of accurate prompt design and suggests that prompting GPT-4 with human-like instructions is not straightforwardly possible for the semantic proximity task. We show that small modifications to the human guidelines already improve the performance, suggesting possible ways for future research.