A negation detection assessment of GPTs: analysis with the xNot360 dataset

TL;DR

This study evaluates GPT models' ability to detect negation in natural language using a custom dataset, revealing significant performance gaps and highlighting limitations in their understanding of negation crucial for high-stakes applications.

Contribution

It introduces the xNot360 dataset and assesses GPT models' negation detection capabilities, exposing their limitations and performance disparities in zero-shot settings.

Findings

GPT-4 outperforms other GPT models in negation detection

GPT models show modest overall proficiency in negation tasks

Performance disparities increase with model complexity

Abstract

Negation is a fundamental aspect of natural language, playing a critical role in communication and comprehension. Our study assesses the negation detection performance of Generative Pre-trained Transformer (GPT) models, specifically GPT-2, GPT-3, GPT-3.5, and GPT-4. We focus on the identification of negation in natural language using a zero-shot prediction approach applied to our custom xNot360 dataset. Our approach examines sentence pairs labeled to indicate whether the second sentence negates the first. Our findings expose a considerable performance disparity among the GPT models, with GPT-4 surpassing its counterparts and GPT-3.5 displaying a marked performance reduction. The overall proficiency of the GPT models in negation detection remains relatively modest, indicating that this task pushes the boundaries of their natural language understanding capabilities. We not only highlight the constraints of GPT models in handling negation but also emphasize the importance of logical reliability in high-stakes domains such as healthcare, science, and law.