CHATTER: A Character Attribution Dataset for Narrative Understanding

TL;DR

The paper introduces CHATTER, a large dataset for evaluating narrative understanding focused on character attributes, providing a benchmark for testing models' comprehension of character development in stories.

Contribution

It creates and validates a comprehensive dataset and benchmark for character attribution in narratives, addressing limitations of existing character taxonomy approaches.

Findings

CHATTER dataset includes 88,124 character-attribute pairs across 2,998 characters.

CHATTEREVAL benchmark correlates well with human annotations.

The dataset enables assessment of narrative understanding and long-context modeling in language models.

Abstract

Computational narrative understanding studies the identification, description, and interaction of the elements of a narrative: characters, attributes, events, and relations. Narrative research has given considerable attention to defining and classifying character types. However, these character-type taxonomies do not generalize well because they are small, too simple, or specific to a domain. We require robust and reliable benchmarks to test whether narrative models truly understand the nuances of the character's development in the story. Our work addresses this by curating the CHATTER dataset that labels whether a character portrays some attribute for 88124 character-attribute pairs, encompassing 2998 characters, 12967 attributes and 660 movies. We validate a subset of CHATTER, called CHATTEREVAL, using human annotations to serve as a benchmark to evaluate the character attribution task in movie scripts. \evaldataset{} also assesses narrative understanding and the long-context modeling capacity of language models.