Cut the CARP: Fishing for zero-shot story evaluation

TL;DR

This paper introduces CARP, a contrastive learning-based zero-shot method for evaluating stories that correlates well with human judgment, reducing the need for annotated datasets or finetuning.

Contribution

The paper presents CARP, a novel contrastive learning approach for zero-shot story evaluation, and introduces the Story-Critique Dataset with 1.3 million story-critique pairs.

Findings

CARP shows strong correlation with human evaluations.

Model outputs outperform finetuning-based methods.

The Story-Critique Dataset is a valuable resource for NLP research.

Abstract

Recent advances in large-scale language models (Raffel et al., 2019; Brown et al., 2020) have brought significant qualitative and quantitative improvements in machine-driven text generation. Despite this, generation and evaluation of machine-generated narrative text remains a challenging problem. Objective evaluation of computationally-generated stories may be prohibitively expensive, require meticulously annotated datasets, or may not adequately measure the logical coherence of a generated story's narratological structure. Informed by recent advances in contrastive learning (Radford et al., 2021), we present Contrastive Authoring and Reviewing Pairing (CARP): a scalable, efficient method for performing qualitatively superior, zero-shot evaluation of stories. We show a strong correlation between human evaluation of stories and those of CARP. Model outputs more significantly correlate with corresponding human input than those language-model based methods which utilize finetuning or prompt engineering approaches. We also present and analyze the Story-Critique Dataset, a new corpora composed of 1.3 million aligned story-critique pairs derived from over 80,000 stories. We expect this corpus to be of interest to NLP researchers.