SAGA: A Participant-specific Examination of Story Alternatives and Goal Applicability for a Deeper Understanding of Complex Events
Sai Vallurupalli, Katrin Erk, Francis Ferraro

TL;DR
This paper introduces a participant-specific approach to analyzing complex narratives by annotating goals and actions, revealing insights into model understanding and the importance of fine-tuning on goal-related data.
Contribution
It presents a novel dataset of goal and action annotations based on participant achievements, and demonstrates the limitations of large language models in capturing goal-driven reasoning.
Findings
Large language models struggle to fully understand goal intentions.
Fine-tuned smaller models outperform larger pre-trained models.
Annotated dataset enables better analysis of goal-based understanding.
Abstract
Interpreting and assessing goal driven actions is vital to understanding and reasoning over complex events. It is important to be able to acquire the knowledge needed for this understanding, though doing so is challenging. We argue that such knowledge can be elicited through a participant achievement lens. We analyze a complex event in a narrative according to the intended achievements of the participants in that narrative, the likely future actions of the participants, and the likelihood of goal success. We collect 6.3K high quality goal and action annotations reflecting our proposed participant achievement lens, with an average weighted Fleiss-Kappa IAA of 80%. Our collection contains annotated alternate versions of each narrative. These alternate versions vary minimally from the "original" story, but can license drastically different inferences. Our findings suggest that while modern…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsSoftware Engineering Techniques and Practices · Systems Engineering Methodologies and Applications · Technology Assessment and Management
