This is not correct! Negation-aware Evaluation of Language Generation Systems

TL;DR

This paper introduces NegBLEURT, a negation-aware evaluation metric for language models, which improves sensitivity to negations by fine-tuning on a specially created negation dataset, outperforming existing metrics.

Contribution

The paper presents a novel negation-aware evaluation metric, NegBLEURT, developed through a rule-based negation tool and fine-tuning, enhancing negation sensitivity in language model evaluations.

Findings

NegBLEURT outperforms existing metrics on negated sentences.

Fine-tuning improves negation sensitivity without harming other performance aspects.

The approach effectively captures negation impacts in language evaluation.

Abstract

Large language models underestimate the impact of negations on how much they change the meaning of a sentence. Therefore, learned evaluation metrics based on these models are insensitive to negations. In this paper, we propose NegBLEURT, a negation-aware version of the BLEURT evaluation metric. For that, we designed a rule-based sentence negation tool and used it to create the CANNOT negation evaluation dataset. Based on this dataset, we fine-tuned a sentence transformer and an evaluation metric to improve their negation sensitivity. Evaluating these models on existing benchmarks shows that our fine-tuned models outperform existing metrics on the negated sentences by far while preserving their base models' performances on other perturbations.