Evaluating prose style transfer with the Bible

TL;DR

This paper introduces a high-quality parallel corpus derived from Bible versions for evaluating prose style transfer systems, along with baseline results to facilitate future research in the area.

Contribution

It provides a standardized, publicly available Bible-based corpus for style transfer evaluation and baseline model results using BLEU and PINC metrics.

Findings

The corpus is highly parallel and well-aligned due to chapter and verse numbering.

Baseline models achieve measurable BLEU and PINC scores.

The corpus is useful for style transfer and potentially other NLP tasks.

Abstract

In the prose style transfer task a system, provided with text input and a target prose style, produces output which preserves the meaning of the input text but alters the style. These systems require parallel data for evaluation of results and usually make use of parallel data for training. Currently, there are few publicly available corpora for this task. In this work, we identify a high-quality source of aligned, stylistically distinct text in different versions of the Bible. We provide a standardized split, into training, development and testing data, of the public domain versions in our corpus. This corpus is highly parallel since many Bible versions are included. Sentences are aligned due to the presence of chapter and verse numbers within all versions of the text. In addition to the corpus, we present the results, as measured by the BLEU and PINC metrics, of several models trained on our data which can serve as baselines for future research. While we present these data as a style transfer corpus, we believe that it is of unmatched quality and may be useful for other natural language tasks as well.