FaNS: a Facet-based Narrative Similarity Metric

TL;DR

FaNS introduces a new narrative similarity metric based on 5W1H facets, leveraging LLMs for granular comparison, significantly outperforming traditional methods in correlating narrative similarities.

Contribution

The paper presents a novel facet-based similarity metric for narratives that improves granularity and accuracy using LLMs and a new dataset.

Findings

FaNS achieves 37% higher correlation than traditional metrics.

Granular facet comparison enhances narrative similarity detection.

Effective in capturing detailed differences between narratives.

Abstract

Similar Narrative Retrieval is a crucial task since narratives are essential for explaining and understanding events, and multiple related narratives often help to create a holistic view of the event of interest. To accurately identify semantically similar narratives, this paper proposes a novel narrative similarity metric called Facet-based Narrative Similarity (FaNS), based on the classic 5W1H facets (Who, What, When, Where, Why, and How), which are extracted by leveraging the state-of-the-art Large Language Models (LLMs). Unlike existing similarity metrics that only focus on overall lexical/semantic match, FaNS provides a more granular matching along six different facets independently and then combines them. To evaluate FaNS, we created a comprehensive dataset by collecting narratives from AllSides, a third-party news portal. Experimental results demonstrate that the FaNS metric exhibits a higher correlation (37\% higher) than traditional text similarity metrics that directly measure the lexical/semantic match between narratives, demonstrating its effectiveness in comparing the finer details between a pair of narratives.