Says Who? Effective Zero-Shot Annotation of Focalization

TL;DR

This paper evaluates the ability of large language models to annotate focalization in literary texts, finding GPT-4o performs comparably to trained humans and offers insights into narrative perspective analysis.

Contribution

It introduces a method for using LLMs to annotate focalization in literature, demonstrating their effectiveness and potential for computational literary analysis.

Findings

GPT-4o achieves an average F1 of 84.79% in focalization annotation.

LLMs' log probabilities reflect annotation difficulty.

Case study on Stephen King novels illustrates practical application.

Abstract

Focalization describes the way in which access to narrative information is restricted or controlled based on the knowledge available to knowledge of the narrator. It is encoded via a wide range of lexico-grammatical features and is subject to reader interpretation. Even trained annotators frequently disagree on correct labels, suggesting this task is both qualitatively and computationally challenging. In this work, we test how well five contemporary large language model (LLM) families and two baselines perform when annotating short literary excerpts for focalization. Despite the challenging nature of the task, we find that LLMs show comparable performance to trained human annotators, with GPT-4o achieving an average F1 of 84.79%. Further, we demonstrate that the log probabilities output by GPT-family models frequently reflect the difficulty of annotating particular excerpts. Finally, we provide a case study analyzing sixteen Stephen King novels, demonstrating the usefulness of this approach for computational literary studies and the insights gleaned from examining focalization at scale.