Developmental Negation Processing in Transformer Language Models

TL;DR

This paper investigates how transformer language models process different types of negation, especially those studied in developmental psychology, using a natural language inference framework to reveal their strengths and limitations.

Contribution

It introduces a novel evaluation of transformer models on negation types from developmental psychology, highlighting their varied reasoning abilities across categories.

Findings

Models perform better on certain negation categories.

Distinct processing patterns are observed across different negation types.

The study reveals limitations in models' understanding of negation.

Abstract

Reasoning using negation is known to be difficult for transformer-based language models. While previous studies have used the tools of psycholinguistics to probe a transformer's ability to reason over negation, none have focused on the types of negation studied in developmental psychology. We explore how well transformers can process such categories of negation, by framing the problem as a natural language inference (NLI) task. We curate a set of diagnostic questions for our target categories from popular NLI datasets and evaluate how well a suite of models reason over them. We find that models perform consistently better only on certain categories, suggesting clear distinctions in how they are processed.