The Self-Contained Negation Test Set

TL;DR

This paper introduces an improved, systematic negation test set for evaluating pretrained language models' understanding of negation, revealing that most models struggle with negation despite some showing expected trends.

Contribution

The paper presents a refined negation test set based on minimal pairs, addressing flaws in previous tests and providing a more controlled evaluation of PLMs' negation interpretation.

Findings

Roberta-large aligns with expectations on negation trends

Bert-base is mostly insensitive to negation

Models often predict semantically forbidden tokens in negation contexts

Abstract

Several methodologies have recently been proposed to evaluate the ability of Pretrained Language Models (PLMs) to interpret negation. In this article, we build on Gubelmann and Handschuh (2022), which studies the modification of PLMs' predictions as a function of the polarity of inputs, in English. Crucially, this test uses ``self-contained'' inputs ending with a masked position: depending on the polarity of a verb in the input, a particular token is either semantically ruled out or allowed at the masked position. By replicating Gubelmann and Handschuh (2022) experiments, we have uncovered flaws that weaken the conclusions that can be drawn from this test. We thus propose an improved version, the Self-Contained Neg Test, which is more controlled, more systematic, and entirely based on examples forming minimal pairs varying only in the presence or absence of verbal negation in English. When applying our test to the roberta and bert base and large models, we show that only roberta-large shows trends that match the expectations, while bert-base is mostly insensitive to negation. For all the tested models though, in a significant number of test instances the top-1 prediction remains the token that is semantically forbidden by the context, which shows how much room for improvement remains for a proper treatment of the negation phenomenon.