On the Limits of Minimal Pairs in Contrastive Evaluation

TL;DR

This paper critically examines the use of minimal pairs in contrastive evaluation of language models, emphasizing the importance of hypothesis motivation and data selection to ensure reliable insights, and proposes a new evaluation suite for machine translation.

Contribution

It highlights conditions necessary for effective contrastive evaluation and introduces a new evaluation suite based on machine-generated text for English-German MT.

Findings

Contrastive evaluation can produce false positives without proper hypothesis motivation.

Using machine-generated text for minimal pairs better approximates deployment conditions.

The proposed evaluation suite improves the reliability of contrastive assessments.

Abstract

Minimal sentence pairs are frequently used to analyze the behavior of language models. It is often assumed that model behavior on contrastive pairs is predictive of model behavior at large. We argue that two conditions are necessary for this assumption to hold: First, a tested hypothesis should be well-motivated, since experiments show that contrastive evaluation can lead to false positives. Secondly, test data should be chosen such as to minimize distributional discrepancy between evaluation time and deployment time. For a good approximation of deployment-time decoding, we recommend that minimal pairs are created based on machine-generated text, as opposed to human-written references. We present a contrastive evaluation suite for English-German MT that implements this recommendation.