MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain

TL;DR

This paper investigates the presence of annotation artifacts in the MedNLI clinical dataset, revealing biases that enable hypothesis-only classifiers to perform well and suggesting improved dataset construction methods for clinical NLP tasks.

Contribution

It identifies specific artifacts in MedNLI and evaluates their impact, offering insights and strategies for creating less biased, more robust clinical NLI datasets.

Findings

Entailed hypotheses contain generic concepts and modifiers.

Neutral hypotheses often include co-occurring or causative conditions.

Adversarial filtering reduces classifier performance on challenging subsets.

Abstract

Crowdworker-constructed natural language inference (NLI) datasets have been found to contain statistical artifacts associated with the annotation process that allow hypothesis-only classifiers to achieve better-than-random performance (Poliak et al., 2018; Gururanganet et al., 2018; Tsuchiya, 2018). We investigate whether MedNLI, a physician-annotated dataset with premises extracted from clinical notes, contains such artifacts (Romanov and Shivade, 2018). We find that entailed hypotheses contain generic versions of specific concepts in the premise, as well as modifiers related to responsiveness, duration, and probability. Neutral hypotheses feature conditions and behaviors that co-occur with, or cause, the condition(s) in the premise. Contradiction hypotheses feature explicit negation of the premise and implicit negation via assertion of good health. Adversarial filtering demonstrates that performance degrades when evaluated on the difficult subset. We provide partition information and recommendations for alternative dataset construction strategies for knowledge-intensive domains.