Constructing a Natural Language Inference Dataset using Generative Neural Networks

TL;DR

This paper introduces neural network-based methods for generating Natural Language Inference datasets, enabling the creation of synthetic data that closely approximates human-crafted datasets and improves classifier performance.

Contribution

It proposes generative neural network models for dataset creation in Natural Language Inference and introduces a new evaluation metric based on classifier accuracy trained on generated data.

Findings

Best generative model's accuracy is within 2.7% of human dataset.

Combining generated and original datasets yields highest classifier accuracy.

Higher ROUGE or METEOR scores do not always correlate with better classification performance.

Abstract

Natural Language Inference is an important task for Natural Language Understanding. It is concerned with classifying the logical relation between two sentences. In this paper, we propose several text generative neural networks for generating text hypothesis, which allows construction of new Natural Language Inference datasets. To evaluate the models, we propose a new metric -- the accuracy of the classifier trained on the generated dataset. The accuracy obtained by our best generative model is only 2.7% lower than the accuracy of the classifier trained on the original, human crafted dataset. Furthermore, the best generated dataset combined with the original dataset achieves the highest accuracy. The best model learns a mapping embedding for each training example. By comparing various metrics we show that datasets that obtain higher ROUGE or METEOR scores do not necessarily yield higher classification accuracies. We also provide analysis of what are the characteristics of a good dataset including the distinguishability of the generated datasets from the original one.