The Self-Contained Negation Test Set
David Kletz (Lattice, LLF - UMR7110, UPCit\'e), Pascal Amsili (Lattice), Marie Candito (LLF UMR7110, UPCit\'e)

TL;DR
This paper introduces an improved, systematic negation test set for evaluating pretrained language models' understanding of negation, revealing that most models struggle with negation despite some showing expected trends.
Contribution
The paper presents a refined negation test set based on minimal pairs, addressing flaws in previous tests and providing a more controlled evaluation of PLMs' negation interpretation.
Findings
Roberta-large aligns with expectations on negation trends
Bert-base is mostly insensitive to negation
Models often predict semantically forbidden tokens in negation contexts
Abstract
Several methodologies have recently been proposed to evaluate the ability of Pretrained Language Models (PLMs) to interpret negation. In this article, we build on Gubelmann and Handschuh (2022), which studies the modification of PLMs' predictions as a function of the polarity of inputs, in English. Crucially, this test uses ``self-contained'' inputs ending with a masked position: depending on the polarity of a verb in the input, a particular token is either semantically ruled out or allowed at the masked position. By replicating Gubelmann and Handschuh (2022) experiments, we have uncovered flaws that weaken the conclusions that can be drawn from this test. We thus propose an improved version, the Self-Contained Neg Test, which is more controlled, more systematic, and entirely based on examples forming minimal pairs varying only in the presence or absence of verbal negation in English.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Attention Is All You Need · Softmax · Linear Layer · Dropout · WordPiece · Residual Connection · Multi-Head Attention · Linear Warmup With Linear Decay · Attention Dropout
