Improving negation detection with negation-focused pre-training

TL;DR

This paper introduces a negation-focused pre-training method that enhances language models' ability to detect negation, improving performance and cross-domain transfer in NLP tasks.

Contribution

It proposes a novel negation-focused pre-training strategy with targeted data augmentation and masking, advancing negation detection in language models.

Findings

Improves negation detection accuracy on benchmark datasets.

Enhances model generalizability across different domains.

Outperforms the baseline NegBERT in experiments.

Abstract

Negation is a common linguistic feature that is crucial in many language understanding tasks, yet it remains a hard problem due to diversity in its expression in different types of text. Recent work has shown that state-of-the-art NLP models underperform on samples containing negation in various tasks, and that negation detection models do not transfer well across domains. We propose a new negation-focused pre-training strategy, involving targeted data augmentation and negation masking, to better incorporate negation information into language models. Extensive experiments on common benchmarks show that our proposed approach improves negation detection performance and generalizability over the strong baseline NegBERT (Khandewal and Sawant, 2020).