A Linguistic Investigation of Machine Learning based Contradiction Detection Models: An Empirical Analysis and Future Perspectives

TL;DR

This paper investigates linguistic challenges faced by machine learning models in natural language inference, highlighting specific syntactic and semantic difficulties and emphasizing the need for linguistically informed training approaches.

Contribution

It provides an empirical analysis of linguistic features affecting model understanding and discusses future directions for improving machine comprehension using external knowledge.

Findings

Models struggle with prepositions and verbs

Difficulty recognizing antonyms and homonyms in context

Incomplete sentences and rare words pose challenges

Abstract

We analyze two Natural Language Inference data sets with respect to their linguistic features. The goal is to identify those syntactic and semantic properties that are particularly hard to comprehend for a machine learning model. To this end, we also investigate the differences between a crowd-sourced, machine-translated data set (SNLI) and a collection of text pairs from internet sources. Our main findings are, that the model has difficulty recognizing the semantic importance of prepositions and verbs, emphasizing the importance of linguistically aware pre-training tasks. Furthermore, it often does not comprehend antonyms and homonyms, especially if those are depending on the context. Incomplete sentences are another problem, as well as longer paragraphs and rare words or phrases. The study shows that automated language understanding requires a more informed approach, utilizing as much external knowledge as possible throughout the training process.