SAGA: A Participant-specific Examination of Story Alternatives and Goal Applicability for a Deeper Understanding of Complex Events

TL;DR

This paper introduces a participant-specific approach to analyzing complex narratives by annotating goals and actions, revealing insights into model understanding and the importance of fine-tuning on goal-related data.

Contribution

It presents a novel dataset of goal and action annotations based on participant achievements, and demonstrates the limitations of large language models in capturing goal-driven reasoning.

Findings

Large language models struggle to fully understand goal intentions.

Fine-tuned smaller models outperform larger pre-trained models.

Annotated dataset enables better analysis of goal-based understanding.

Abstract

Interpreting and assessing goal driven actions is vital to understanding and reasoning over complex events. It is important to be able to acquire the knowledge needed for this understanding, though doing so is challenging. We argue that such knowledge can be elicited through a participant achievement lens. We analyze a complex event in a narrative according to the intended achievements of the participants in that narrative, the likely future actions of the participants, and the likelihood of goal success. We collect 6.3K high quality goal and action annotations reflecting our proposed participant achievement lens, with an average weighted Fleiss-Kappa IAA of 80%. Our collection contains annotated alternate versions of each narrative. These alternate versions vary minimally from the "original" story, but can license drastically different inferences. Our findings suggest that while modern large language models can reflect some of the goal-based knowledge we study, they find it challenging to fully capture the design and intent behind concerted actions, even when the model pretraining included the data from which we extracted the goal knowledge. We show that smaller models fine-tuned on our dataset can achieve performance surpassing larger models.