Large language models struggle with ethnographic text annotation

TL;DR

Large language models currently cannot reliably automate ethnographic text annotation, especially for complex or ambiguous features, and still require human expertise for accurate cross-cultural research analysis.

Contribution

This study systematically evaluates the limitations of state-of-the-art LLMs in ethnographic annotation tasks, highlighting their current performance gaps compared to human coders.

Findings

LLMs perform poorly on complex and ambiguous ethnographic features

Human inter-coder reliability sets an upper bound for LLM accuracy

Models fall short even on features with high human agreement

Abstract

Large language models (LLMs) have shown promise for automated text annotation, raising hopes that they might accelerate cross-cultural research by extracting structured data from ethnographic texts. We evaluated 7 state-of-the-art LLMs on their ability to annotate 121 ritual features across 567 ethnographic excerpts. Performance was limited, falling well below levels required for reliable automated annotation. Longer texts, features requiring ordinal distinctions, and ambiguous constructs proved particularly difficult. Human inter-coder reliability set an approximate ceiling on LLM accuracy: features that human coders found difficult to agree upon were also difficult for LLMs. Yet even on features where humans reliably agreed, models fell short of human performance. Our findings suggest that LLMs cannot yet substitute for human expertise in ethnographic annotation.