Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models

TL;DR

This paper investigates how prompt selection influences the accuracy of text annotations in social sciences using large language models, demonstrating that optimized prompts significantly improve performance and providing a tool for prompt optimization.

Contribution

It introduces an automatic prompt optimization method for large language models in social science text annotation tasks and offers a practical browser-based implementation.

Findings

Prompt choice greatly affects annotation accuracy.

Optimized prompts outperform standard prompts.

The method is accessible via a user-friendly online tool.

Abstract

Large Language Models have recently been applied to text annotation tasks from social sciences, equalling or surpassing the performance of human workers at a fraction of the cost. However, no inquiry has yet been made on the impact of prompt selection on labelling accuracy. In this study, we show that performance greatly varies between prompts, and we apply the method of automatic prompt optimization to systematically craft high quality prompts. We also provide the community with a simple, browser-based implementation of the method at https://prompt-ultra.github.io/ .