Classifying Unreliable Narrators with Large Language Models

TL;DR

This paper explores using large language models to identify unreliable narrators in various texts, leveraging a new annotated dataset and different learning strategies, revealing the task's complexity and potential.

Contribution

It introduces TUNa, a novel dataset for unreliable narrator classification, and evaluates LLMs' performance across multiple textual domains and learning paradigms.

Findings

LLMs face challenges in reliably classifying unreliable narrators

Few-shot and fine-tuning improve classification performance

The dataset and code are publicly released for future research

Abstract

Often when we interact with a first-person account of events, we consider whether or not the narrator, the primary speaker of the text, is reliable. In this paper, we propose using computational methods to identify unreliable narrators, i.e. those who unintentionally misrepresent information. Borrowing literary theory from narratology to define different types of unreliable narrators based on a variety of textual phenomena, we present TUNa, a human-annotated dataset of narratives from multiple domains, including blog posts, subreddit posts, hotel reviews, and works of literature. We define classification tasks for intra-narrational, inter-narrational, and inter-textual unreliabilities and analyze the performance of popular open-weight and proprietary LLMs for each. We propose learning from literature to perform unreliable narrator classification on real-world text data. To this end, we experiment with few-shot, fine-tuning, and curriculum learning settings. Our results show that this task is very challenging, and there is potential for using LLMs to identify unreliable narrators. We release our expert-annotated dataset and code and invite future research in this area.