Fabula Entropy Indexing: Objective Measures of Story Coherence
Louis Castricato, Spencer Frazier, Jonathan Balloch, Mark Riedl

TL;DR
This paper introduces Fabula Entropy Indexing, a novel objective evaluation method for story coherence based on human agreement in question-answering, addressing the challenge of assessing narrative quality in automated story generation.
Contribution
It proposes two entropy-based metrics, EWC and ETC, grounded in theory, to objectively measure global and local story coherence, validated through human studies.
Findings
Entropy indices reliably measure story coherence.
Metrics distinguish between coherent and incoherent stories.
Validated on human-written and corrupted stories.
Abstract
Automated story generation remains a difficult area of research because it lacks strong objective measures. Generated stories may be linguistically sound, but in many cases suffer poor narrative coherence required for a compelling, logically-sound story. To address this, we present Fabula Entropy Indexing (FEI), an evaluation method to assess story coherence by measuring the degree to which human participants agree with each other when answering true/false questions about stories. We devise two theoretically grounded measures of reader question-answering entropy, the entropy of world coherence (EWC), and the entropy of transitional coherence (ETC), focusing on global and local coherence, respectively. We evaluate these metrics by testing them on human-written stories and comparing against the same stories that have been corrupted to introduce incoherencies. We show that in these…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
