PASTA: A Dataset for Modeling Participant States in Narratives

TL;DR

This paper introduces PASTA, a new dataset for modeling participant states in narratives, enabling reasoning about implicit states and their impact on story understanding, with experiments showing current models have room for improvement.

Contribution

The paper presents PASTA, a novel crowdsourced dataset with reasoning tasks for participant states in narratives, facilitating research on implicit state inference and story revision.

Findings

LLMs can partially reason about participant states

Significant room for improvement in reasoning with diverse knowledge types

New tasks challenge models to infer, revise, and explain state changes

Abstract

The events in a narrative are understood as a coherent whole via the underlying states of their participants. Often, these participant states are not explicitly mentioned, instead left to be inferred by the reader. A model that understands narratives should likewise infer these implicit states, and even reason about the impact of changes to these states on the narrative. To facilitate this goal, we introduce a new crowdsourced English-language, Participant States dataset, PASTA. This dataset contains inferable participant states; a counterfactual perturbation to each state; and the changes to the story that would be necessary if the counterfactual were true. We introduce three state-based reasoning tasks that test for the ability to infer when a state is entailed by a story, to revise a story conditioned on a counterfactual state, and to explain the most likely state change given a revised story. Experiments show that today's LLMs can reason about states to some degree, but there is large room for improvement, especially in problems requiring access and ability to reason with diverse types of knowledge (e.g. physical, numerical, factual).