Can neural networks understand monotonicity reasoning?

TL;DR

This paper introduces the Monotonicity Entailment Dataset (MED) to evaluate neural models' ability to perform monotonicity reasoning, revealing significant performance gaps and limitations in generalization, especially on downward reasoning.

Contribution

The paper creates a new comprehensive dataset for monotonicity reasoning and analyzes neural models' performance, highlighting their limitations and the need for improved reasoning capabilities.

Findings

State-of-the-art models perform below 55% on MED

Models struggle particularly with downward monotonic reasoning

Data augmentation reveals limited generalization in models

Abstract

Monotonicity reasoning is one of the important reasoning skills for any intelligent natural language inference (NLI) model in that it requires the ability to capture the interaction between lexical and syntactic structures. Since no test set has been developed for monotonicity reasoning with wide coverage, it is still unclear whether neural models can perform monotonicity reasoning in a proper way. To investigate this issue, we introduce the Monotonicity Entailment Dataset (MED). Performance by state-of-the-art NLI models on the new test set is substantially worse, under 55%, especially on downward reasoning. In addition, analysis using a monotonicity-driven data augmentation method showed that these models might be limited in their generalization ability in upward and downward reasoning.