FicSim: A Dataset for Multi-Faceted Semantic Similarity in Long-Form Fiction

TL;DR

FicSim introduces a novel dataset of long-form fiction with multi-faceted similarity annotations, enabling better evaluation of language models for literary analysis tasks, addressing limitations of existing datasets.

Contribution

The paper presents FICSIM, a new dataset with detailed similarity scores for long-form fiction, and evaluates embedding models, highlighting their focus on surface features over semantic understanding.

Findings

Models tend to focus on surface-level features.

FICSIM enables nuanced evaluation of semantic similarity.

Dataset creation involved author consent and expert validation.

Abstract

As language models become capable of processing increasingly long and complex texts, there has been growing interest in their application within computational literary studies. However, evaluating the usefulness of these models for such tasks remains challenging due to the cost of fine-grained annotation for long-form texts and the data contamination concerns inherent in using public-domain literature. Current embedding similarity datasets are not suitable for evaluating literary-domain tasks because of a focus on coarse-grained similarity and primarily on very short text. We assemble and release FICSIM, a dataset of long-form, recently written fiction, including scores along 12 axes of similarity informed by author-produced metadata and validated by digital humanities scholars. We evaluate a suite of embedding models on this task, demonstrating a tendency across models to focus on surface-level features over semantic categories that would be useful for computational literary studies tasks. Throughout our data-collection process, we prioritize author agency and rely on continual, informed author consent.