LLMs for automatic annotation of Mandarin narrative transcripts

TL;DR

This study assesses the effectiveness of Large Language Models in automating discourse-level annotation of Mandarin narratives, demonstrating promising accuracy and efficiency gains but also highlighting ongoing challenges.

Contribution

It provides the first systematic evaluation of LLMs for macrostructure annotation in non-English spoken narratives, with open-sourced prompt templates.

Findings

Best LLM achieved agreement with human raters (k=.794)

Annotation time reduced by 65% with LLMs

Model reliability decreased on narratives with greater lexical variation

Abstract

Linguistic annotation of transcribed speech is essential for research in language acquisition, language disorders, and sociolinguistics, yet remains labor-intensive and time-consuming. While Large Language Models (LLMs) have shown promise in automating annotation tasks, their ability to handle complex discourse-level annotation in non-English languages remains understudied. This study evaluates whether LLMs can reliably annotate narrative macrostructure-the hierarchical organization of story grammar elements-in spoken Mandarin, using the Multilingual Assessment Instrument for Narratives (MAIN) as a testbed. We compared four LLMs against trained human annotators on narratives produced by children, young adults, and older adults. The best-performing model achieved agreement with human raters (k=.794) approaching human-human reliability levels (k=.872) while reducing annotation time by 65%, whereas the locally deployable lightweight model performed substantially worse. Annotation difficulty varied systematically by macrostructure element type, with categories requiring subtle semantic differentiation posing persistent challenges. Furthermore, model reliability decreased on young adult narratives, which exhibited greater lexical variation, semantic ambiguity, and multi-element integration within single utterances. These findings suggest that LLMs can effectively support discourse-level annotation in non-English spoken corpora, while highlighting the continued need for human oversight in semantically complex tasks. Our prompt templates are open sourced for future use.