ARN: Analogical Reasoning on Narratives
Zhivar Sourati, Filip Ilievski, Pia Sommerauer, Yifan Jiang

TL;DR
This paper introduces a new benchmark for evaluating large language models' ability to perform analogical reasoning on narratives, revealing current limitations and potential improvements through guided reasoning.
Contribution
It develops a comprehensive framework and benchmark for narrative analogical reasoning, extending word-based analogies to complex narrative systems and evaluating LLM performance.
Findings
LLMs recognize near but not far analogies
GPT-4 struggles with far analogies in zero-shot
Chain-of-thought improves reasoning performance
Abstract
As a core cognitive skill that enables the transferability of information across domains, analogical reasoning has been extensively studied for both humans and computational models. However, while cognitive theories of analogy often focus on narratives and study the distinction between surface, relational, and system similarities, existing work in natural language processing has a narrower focus as far as relational analogies between word pairs. This gap brings a natural question: can state-of-the-art large language models (LLMs) detect system analogies between narratives? To gain insight into this question and extend word-based relational analogies to relational system analogies, we devise a comprehensive computational framework that operationalizes dominant theories of analogy, using narrative elements to create surface and system mappings. Leveraging the interplay between these…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques
MethodsFocus
