Several Experiments on Investigating Pretraining and Knowledge-Enhanced Models for Natural Language Inference

TL;DR

This paper investigates the effects of pretraining and external knowledge integration on natural language inference, aiming to understand their roles and effectiveness in improving NLI models.

Contribution

It provides a comparative analysis of pretraining and external knowledge sources, revealing how each contributes to NLI performance and understanding.

Findings

Pretraining helps NLI by learning inference-related knowledge.

External knowledge sources can improve NLI but in different ways than pretraining.

Certain artifacts in data annotation may influence model performance.

Abstract

Natural language inference (NLI) is among the most challenging tasks in natural language understanding. Recent work on unsupervised pretraining that leverages unsupervised signals such as language-model and sentence prediction objectives has shown to be very effective on a wide range of NLP problems. It would still be desirable to further understand how it helps NLI; e.g., if it learns artifacts in data annotation or instead learn true inference knowledge. In addition, external knowledge that does not exist in the limited amount of NLI training data may be added to NLI models in two typical ways, e.g., from human-created resources or an unsupervised pretraining paradigm. We runs several experiments here to investigate whether they help NLI in the same way, and if not,how?