Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering

TL;DR

This paper systematically compares various neural network models for sentence pair tasks across multiple datasets, highlighting key design factors and providing insights into model performance and dataset size effects.

Contribution

It offers a comprehensive evaluation of neural network architectures for sentence pair modeling, revealing critical factors and challenging previous claims about Tree-LSTM effectiveness.

Findings

Encoding contextual info by LSTM is crucial.

Inter-sentence interactions significantly improve performance.

Tree-LSTM's benefits are dataset-dependent.

Abstract

In this paper, we analyze several neural network designs (and their variations) for sentence pair modeling and compare their performance extensively across eight datasets, including paraphrase identification, semantic textual similarity, natural language inference, and question answering tasks. Although most of these models have claimed state-of-the-art performance, the original papers often reported on only one or two selected datasets. We provide a systematic study and show that (i) encoding contextual information by LSTM and inter-sentence interactions are critical, (ii) Tree-LSTM does not help as much as previously claimed but surprisingly improves performance on Twitter datasets, (iii) the Enhanced Sequential Inference Model is the best so far for larger datasets, while the Pairwise Word Interaction Model achieves the best performance when less data is available. We release our implementations as an open-source toolkit.