Multi-turn Inference Matching Network for Natural Language Inference

TL;DR

The paper introduces MIMN, a multi-turn inference model for NLI that processes different matching features separately and uses memory to improve interaction, achieving state-of-the-art results.

Contribution

It proposes a novel multi-turn inference framework with memory for NLI, focusing on separate matching features to enhance inference accuracy.

Findings

Outperforms existing models on three NLI datasets.

Achieves state-of-the-art performance across all tested datasets.

Demonstrates the effectiveness of multi-turn inference with memory in NLI.

Abstract

Natural Language Inference (NLI) is a fundamental and challenging task in Natural Language Processing (NLP). Most existing methods only apply one-pass inference process on a mixed matching feature, which is a concatenation of different matching features between a premise and a hypothesis. In this paper, we propose a new model called Multi-turn Inference Matching Network (MIMN) to perform multi-turn inference on different matching features. In each turn, the model focuses on one particular matching feature instead of the mixed matching feature. To enhance the interaction between different matching features, a memory component is employed to store the history inference information. The inference of each turn is performed on the current matching feature and the memory. We conduct experiments on three different NLI datasets. The experimental results show that our model outperforms or achieves the state-of-the-art performance on all the three datasets.

Figures

Equations

$${\bar{p}}_i={\text{BiLSTM}}_{enc}(p,i),$$

$${\bar{q}}_j={\text{BiLSTM}}_{enc}(q,j),$$

$$e_{ij}={\bar{p}}_i^T{\bar{q}}_j,$$

$${\tilde{p}}_i=\sum _{j=1}^{l_q}\frac{exp(e_{ij})}{\sum {}_{t=1}^{l_q}exp(e_{it})}{\bar{q}}_j,$$

$$u_{p,i}^c=f^c(\overline{p_i},{\tilde{p}}_i)$$

$$u_{p,i}^s=f^s(\overline{p_i},{\tilde{p}}_i)$$

$$u_{p,i}^m=f^m(\overline{p_i},{\tilde{p}}_i)$$

$$u_p=[u_p^1,u_p^2,u_p^3]=[u_p^c,u_p^s,u_p^m],$$

$$c_{p,i}^k={\text{BiLSTM}}_{inf}(W_{inf}[u_{p,i}^k;m_{p,i}^{(k-1)}]),$$

$$m_{p,i}^k=g\odot c_{p,i}^k+(1-g)\odot m_{p,i}^{(k-1)},$$

$$g=\sigma (W_g[c_{p,i}^k;m_{p,i}^{(k-1)}]+b_g),$$

$$c_{p,i}^k={\text{BiLSTM}}_{inf}(W_{inf}[u_{p,i}^k]),$$

$$m_p^{inf}=[\{c_{p,i}^1{\}}_1^{l_p};\{c_{p,i}^2{\}}_1^{l_p};\{c_{p,i}^3{\}}_1^{l_p}].$$

$$m_{p,i}^k=\text{ReLU}(W_m[c_{p,i}^k;m_{p,i}^{(k-1)}]).$$

Full text

1 Introduction

Natural Language Inference (NLI) is a crucial subtopic in Natural Language Processing (NLP). Most studies treat NLI as a classification problem, aiming at recognizing the relation types of hypothesis-premise sentence pairs, usually including “Entailment”, “Contradiction” and “Neutral”.

NLI is also called Recognizing Textual Entailment (RTE) [7] in earlier works and a lot of statistical-based [9] and rule-based approaches [19] are proposed to solve the problem. In 2015, Bowman released the SNLI corpus [3] that provides more than 570K hypothesis-premise sentence pairs. The large-scale data of SNLI allows a Neural Network (NN) based model to perform on the NLI. Since then, a variety of NN based models have been proposed, most of which can be divided into two kinds of frameworks. The first one is based on “Siamense” network [3, 22]. It first applies either Recurrent Neural Network (RNN) or Convolutional Neural Networks (CNN) to generates sentence representations on both premise and hypothesis, and then concatenate them for the final classification. The second one is called “matching-aggregation” network [33, 36]. It matches two sentences at word level, and then aggregates the matching results to generate a fixed vector for prediction. Matching is implemented by several functions based on element-wise operations [34, 25]. Studies on SNLI show that the second one performs better.

Though the second framework has made considerable success on the NLI task, there are still some limitations. First, the inference on the mixed matching feature only adopts one-pass process, which means some detailed information would not be retrieved once missing. While the multi-turn inference can overcome this deficiency and make better use of these matching features. Second, the mixed matching feature only concatenates different matching features as the input for aggregation. It lacks interaction among various matching features. Furthermore, it treats all the matching features equally and cannot assign different importance to different matching features.

In this paper, we propose the MIMN model to tackle these limitations. Our model uses the matching features described in [33, 5]. However, we do not simply concatenate the features but introduce a multi-turn inference mechanism to infer different matching features with a memory component iteratively. The merits of MIMN are as follows:

• $\bullet$

MIMN first matches two sentences from various perspectives to generate different matching features and then aggregates these matching features by multi-turn inference mechanism. During the multi-turn inference, each turn focuses on one particular matching feature, which helps the model extract the matching information adequately.

• $\bullet$

MIMN establishes the contact between the current and previous matching features through memory component. The memory component store the inference message of the previous turn. In this way, the inference information flows.

We conduct experiments on three NLI datasets: SNLI [3], SCITAIL [12] and MPE [14]. On the SNLI dataset, our single model achieves 88.3% in accuracy and our ensemble model achieves 89.3% in terms of accuracy, which are both comparable with the state-of-the-art results. Furthermore, our MIMN model outperforms all previous works on both SCITAIL and MPE dataset. Especially, the model gains substantial (8.9%) improvement on MPE dataset which contains multiple premises. This result shows our model is expert in aggregating the information of multiple premises.

2 Related Work

Early work on the NLI task mainly uses conventional statistical methods on small-scale datasets [7, 20]. Recently, the neural models on NLI are based on large-scale datasets and can be categorized into two central frameworks: (i) Siamense-based framework which focuses on building sentence embeddings separately and integrates the two sentence representations to make the final prediction [22, 17, 23, 4, 29, 24, 32]; (ii) “matching-aggregation” framework which uses various matching methods to get the interactive space of two input sentences and then aggregates the matching results to dig for deep information [27, 15, 16, 28, 25, 36, 21, 38, 32, 10, 8].

Our model is directly motivated by the approaches proposed by [34, 5]. [34] introduces the “matching-aggregation” framework to compare representations between words and then aggregate their matching results for final decision.

[5] enhances the comparing approaches by adding element-wise subtraction and element-wise multiplication, which further improve the performance on SNLI. The previous work shows that matching layer is an essential component of this framework and different matching methods can affect the final classification result.

Various attention-based memory neural networks [37] have been explored to solve the NLI problem [15, 6, 23]. [15] presents a model of deep fusion LSTMs (DF-LSTMs) (Long Short-Term Memory ) which utilizes a strong interaction between text pairs in a recursive matching memory. [6] uses a memory network to extend the LSTM architecture. [23] employs a variable sized memory model to enrich the LSTM-based input encoding information. However, all the above models are not specially designed for NLI and they all focus on input sentence encoding.

Inspired by the previous work, we propose the MIMN model. We iteratively update memory by feeding in different sequence matching features. We are the first to apply memory mechanism to matching component for the NLI task. Our experiment results on several datasets show that our MIMN model is significantly better than the previous models.

3 Model

In this section, we describe our MIMN model, which consists of the following five major components: encoding layer, attention layer, matching layer, multi-turn inference layer and output layer. Fig. 1 shows the architecture of our MIMN model.

We represent each example of the NLI task as a triple $(p,q,y)$, where $p=[p_{1},p_{2},\\ \cdots,p_{l_{p}}]$ is a given premise, $q=[q_{1},q_{2},\cdots,q_{l_{q}}]$ is a given hypothesis, $p_{i}$ and $q_{j}\in\mathbb{R}^{r}$ are word embeddings of r-dimension. The true label $y\in\mathcal{Y}$ indicates the logical relationship between the premise $p$ and the hypothesis $q$, where $\mathcal{Y}=\{neutral,entailment,\\ contradiction\}$. Our model aims to compute the conditional probability $Pr(y|p,q)$ and predict the label for examples in testing data set by $y^{*}=argmax_{y\in\mathcal{Y}}Pr(y|p,q)$.

3.1 Encoding Layer

In this paper, we utilize a bidirectional LSTM (BiLSTM) [11] as our encoder to transform the word embeddings of premise and hypothesis to context vectors. The premise and the hypothesis share the same weights of BiLSTM.

[TABLE]

where the context vectors $\bar{p}_{i}$ and $\bar{q}_{j}$ are the concatenation of the forward and backward hidden outputs of BiLSTM respectively. The outputs of the encoding layer are the context vectors $p\in\mathbb{R}^{l_{p}\times 2d}$ and $q\in\mathbb{R}^{l_{q}\times 2d}$, where $d$ is the number of hidden units of $\text{BiLSTM}_{enc}$.

3.2 Attention Layer

On the NLI task, the relevant contexts between the premise and the hypothesis are important clues for final classification. The relevant contexts can be acquired by a soft-attention mechanism [2, 18], which has been applied to a bunch of tasks successfully. The alignments between a premise and a hypothesis are based on a score matrix. There are three most commonly used methods to compute the score matrix: linear combination, bilinear combination, and dot product. For simplicity, we choose dot product in the following computation [25]. First, each element in the score matrix is computed based on the context vectors of $\bar{p}_{i}$ and $\bar{q}_{j}$ as follows:

[TABLE]

where $\bar{p}_{i}$ and $\bar{q}_{j}$ are computed in Equations (1) and (2), and $e_{ij}$ is a scalar which indicates how $\bar{p}_{i}$ is related to $\bar{q}_{j}$.

Then, we compute the alignment vectors for each word in the premise and the hypothesis as follows:

[TABLE]

where $\tilde{p_{i}}\in\mathbb{R}^{2d}$ is the weighted summaries of thehypothesis in terms of each word in the premise. The same operation is applied to $\tilde{q_{j}}\in\mathbb{R}^{2d}$. The outputs of this layer are $\tilde{p_{i}}\in\mathbb{R}^{l_{p}\times 2d}$ and $\tilde{q_{j}}\in\mathbb{R}^{l_{q}\times 2d}$. For the context vectors $\bar{p}$, the relevant contexts in the hypothesis $\bar{q}$ are represented in $\tilde{p}$. The same is applied to $\bar{q}$ and $\tilde{q}$.

3.3 Matching Layer

The goal of the matching layer is to match the context vectors $\bar{p}$ and $\bar{q}$ with the corresponding aligned vectors $\tilde{p}$ and $\tilde{q}$ from multi-perspective to generate a matching sequence.

In this layer, we match each context vector $p_{i}$ against each aligned vector $\tilde{p}_{i}$ to capture richer semantic information. We design three effective matching functions: $f^{c}$, $f^{s}$ and $f^{m}$ to match two vectors [31, 33, 5]. Each matching function takes the context vector $\bar{p}_{i}$ ($\bar{q}_{j}$) and the aligned vector $\tilde{p}_{i}$ ($\tilde{q}_{j}$) as inputs, then matches the inputs by an feed-forward network based on a particular matching operation and finally outputs a matching vector. The formulas of the three matching functions $f^{c}$, $f^{s}$ and $f^{m}$ are described in formulas (5) (6) (7). To avoid repetition, we will only describe the application of these functions to $\bar{p}$ and $\tilde{p}$. The readers can infer these equations for $\bar{q}$ and $\tilde{q}$.

[TABLE]

where $;\,$, $-$, and $\odot$ represent concatenation, subtraction, and multiplication respectively, $W^{c}\in\mathbb{R}^{4d\times d}$, $W^{s}\in\mathbb{R}^{2d\times d}$ and $W^{m}\in\mathbb{R}^{2d\times d}$ are weight parameters to be learned, and $b^{c},b^{s},b^{m}\in\mathbb{R}^{d}$ are bias parameters to be learned. The outputs of each matching function are $u_{p,i}^{c},u_{p,i}^{s},u_{p,i}^{m}\in\mathbb{R}^{d}$, which represent the matching result from three perspectives respectively. After matching the context vectors $\bar{p}$ and the aligned vectors $\tilde{p}$ by $f^{c}$, $f^{s}$ and $f^{m}$, we can get three matching features $u_{p}^{c}=\{u_{p,i}^{c}\}^{l_{p}}_{1}$, $u_{p}^{s}=\{u_{p,i}^{s}\}^{l_{p}}_{1}$ and $u_{p}^{m}=\{u_{p,i}^{m}\}^{l_{p}}_{1}$.

The $u_{p}^{c}$ can be considered as a joint-feature of combing the context vectors $\bar{p}$ with aligned vectors $\tilde{p}$, which preserves all the information. And the $u_{p}^{s}$ can be seen as a diff-feature of the $\bar{p}$ and $\tilde{p}$, which preserves the different parts and removes the similar parts. And the $u_{p}^{m}$ can be regarded as a sim-feature of $p$ and $\bar{p}$, which emphasizes on the similar parts and neglects the different parts between $bar{p}$ and $\tilde{p}$. Each feature helps us focus on particular parts between the context vectors and the aligned vectors. These matching features are vector representations with low dimension, but containing high-order semantic information. To make further use of these matching features, we collect them to generate a matching sequence $u_{p}$.

[TABLE]

where $u_{p}^{1},u_{p}^{2},u_{p}^{3}\in\mathbb{R}^{l_{p}\times d}$.

The output of this layer is the matching sequence $u_{p}$, which stores three kinds of matching features. The order of the matching features in $u_{p}$ is inspired by the attention trajectory of human beings making inference on premise and hypothesis. We process the matching sequence in turn in the multi-turn inference layer. Intuitively, given a premise and a hypothesis, we will first read the original sentences to find the relevant information. Next, it’s natural for us to combine all the parts of the original information and the relevant information. Then we move the attention to the different parts. Finally, we pay attention to the similar parts.

3.4 Multi-turn Inference Layer

In this layer, we aim to acquire inference outputs by aggregating the information in the matching sequence by multi-turn inference mechanism. We regard the inference on the matching sequence as the multi-turn interaction among various matching features. In each turn, we process one matching feature instead of all the matching features [5, 8]. To enhance the information interaction between matching features, a memory component is employed to store the inference information of the previous turns. Then, the inference of each turn is based on the current matching feature and the memory. Here, we utilize another BiLSTM for the inference.

[TABLE]

where $c^{k}_{p,i}\in\mathbb{R}^{2d}$ is an inference vector in the current turn, $k=[1,2,3]$ is the index current turn, $i=[1,2,3,\cdots,l_{p}]$, $m^{(k-1)}_{p,i}\in\mathbb{R}^{2d}$ is a memory vector stores the historical inference information, and $W_{inf}\in\mathbb{R}^{3d\times d}$ is used for dimension reduction.

Then we update the memory by combining the current inference vector $c^{k}_{p,i}$ with the memory vector of last turn $m^{(k-1)i}_{p}$. An update gate is used to control the ratio of current information and history information adaptively [35]. The initial values of all the memory vectors are all zeros.

[TABLE]

where $W_{g}\in\mathbb{R}^{4d\times 2d}$ and $b_{g}\in\mathbb{R}^{2d}$ are parameters to be learned, and $\sigma$ is a sigmoid function to compress the ratio between 0-1. Finally, we use the latest memory matrix $\{m^{3i}_{p}\}_{1}^{l_{p}}$ as the inference output of premise $m^{inf}_{p}$. Then we calculate $m^{inf}_{q}$ in a similar way. The final outputs of this layer are $m^{inf}_{p}$ and $m^{inf}_{q}$.

3.5 Output Layer

The final relationship judgment depends on the sentence embeddings of premise and hypothesis. We convert $m^{inf}_{p}$ and $m^{inf}_{q}$ to sentence embeddings of premise and hypothesis by max pooling and average pooling. Next, we concatenate the two sentence embeddings to a fixed-length output vector. Then we feed the output vector to a multilayer perceptron (MLP) classifier that includes a hidden layer with $tanh$ activation and a softmax layer to get the final prediction. The model is trained end-to-end. We employ multi-class cross-entropy as the cost function when training the model.

4 Experiment

4.1 Data

To verify the effectiveness of our model, we conduct experiments on three NLI datasets. The basic information about the three datasets is shown in Table 1.

The large SNLI [3] corpus is served as a major benchmark for the NLI task. The MPE corpus [14] is a newly released textual entailment dataset. Each pair in MPE consists of four premises, one hypothesis, and one label, which is different from the standard NLI datasets. Entailment relationship holds if the hypothesis comes from the same image as the four premises. The SCITAIL [12] is a dataset about science question answering. The premises are created from relevant web sentences, while hypotheses are created from science questions and the corresponding answer candidates.

4.2 Models for Comparison

We compare our model with “matching-aggregation” related and attention-based memory related models. In addition, to verify the effectiveness of these major components in our model, we design the following model variations for comparison:

• $\bullet$

**ESIM ** We choose the ESIM model as our baseline. It mixes all the matching feature together in the matching layer and then infers the matching result in a single-turn with a BiLSTM.

• $\bullet$

600D MIMN: This is our main model described in section 3.

• $\bullet$

600D MIMN-memory: This model removes the memory component. The motivation of this experiment is to verify whether the multiple turns inference can acquire more sufficient information than one-pass inference. In this model, we process one matching feature in one iteration. The three matching features are encoded by $\text{BiLSTM}_{inf}$ in multi-turns iteratively without previous memory information. The output of each iteration is concatenated to be the final output of the multi-turn inference layer:

[TABLE]

• $\bullet$

**600D MIMN-gate+ReLU **: This model replaces the update gate in the memory component with a ReLU layer. The motivation of this model is to verify the effectiveness of update gate for combining current inference result and previous memory. Then the Equation (10) is changed into Equation (13).

[TABLE]

4.3 Experimental Settings

We implement our model with Tensorflow [1]. We initialize the word embeddings by the pre-trained embeddings of 300D GloVe 840B vectors [26]. The word embeddings of the out-of-vocabulary words are randomly initialized. The hidden units of $\text{BiLSTM}_{enc}$ and $\text{BiLSTM}_{inf}$ are 300 dimensions. All weights are constrained by L2 regularization with the weight decay coefficient of 0.0003. We also apply dropout [30] to all the layers with a dropout rate of 0.2. Batch size is set to 32. The model is optimized with Adam [13] with an initial learning rate of 0.0005, the first momentum of 0.9 and the second of 0.999. The word embeddings are fixed during all the training time. We use early-stopping (patience=10) based on the validation set accuracy. We use three turns on all the datasets. The evaluation metric is the classification accuracy. To help duplicate our results, we will release our source code at https://github.com/blcunlp/RTE/tree/master/MIMN.

4.4 Experiments on SNLI

Experimental results of the current state-of-the-art models and three variants of our model are listed in Table 2. The first group of models (1)-(3) are the attention-based memory models on the NLI task. [15] uses external memory to increase the capacity of LSTMs. [23] utilizes an encoding memory matrix to maintain the input information. [6] extends the LSTM architecture with a memory network to enhance the interaction between the current input and all previous inputs.

The next group of models (4)-(12) belong to the “matching-aggregation” framework with bidirectional inter-attention. Decomposable attention [25] first applies the “matching-aggregation” on SNLI dataset explicitly. [33] enriches the framework with several comparison functions. BiMPM [36] employs a multi-perspective matching function to match the two sentences.

ESIM [5] further sublimates the framework by enhancing the matching tuples with element-wise subtraction and element-wise multiplication. ESIM achieves 88.0% in accuracy on the SNLI test set, which exceeds the human performance (87.7%) for the first time. [32] and [9] both further improve the performance by taking the ESIM model as a baseline model. The studies related to “matching-aggregation” but without bidirectional interaction are not listed [27, 34].

Motivated by the attention-based memory models and the bidirectional inter-attention models, we propose the MIMN model. The last group of models (13)-(16) are models described in this paper. Our single MIMN model obtains an accuracy of 88.3% on SNLI test set, which is comparable with the current state-of-the-art single models. The single MIMN model improves 0.3% on the test set compared with ESIM, which shows that multi-turn inference based on the matching features and memory achieves better performance. From model (14), we also observe that memory is generally beneficial, and the accuracy drops 0.8% when the memory is removed. This finding proves that the interaction between matching features is significantly important for the final classification. To explore the way of updating memory, we replace the update gate in MIMN with a ReLU layer to update the memory, which drops 0.1%.

To further improve the performance, an ensemble model MIMN is built for comparison. We design the ensemble model by simply averaging the probability distributions [36] of four MIMN models. Each of the models has the same architecture but initialized by different seeds. Our ensemble model achieves the state-of-the-art performance by obtains an accuracy of 89.3% on SNLI test set.

4.5 Experiments on MPE

The MPE dataset is a brand-new dataset for NLI with four premises, one hypothesis, and one label. In order to maintain the same data format as other textual entailment datasets (one premise, one hypothesis, and one label), we concatenate the four premises as one premise.

Table 4 shows the results of our models along with the published models on this dataset. LSTM is a conditional LSTM model used in [27]. WbW-Attention aligns each word in the hypothesis with the premise. The state-of-the-art model on MPE dataset is SE model proposed by [14], which makes four independent predictions for each sentence pairs, and the final prediction is the summation of four predictions. Compared with SE, our MIMN model obtains a dramatic improvement (9.7%) on MPE dataset by achieving 66.0% in accuracy.

To compare with the bidirectional inter-attention model, we re-implement the ESIM, which obtains 59.0% in accuracy. We observe that MIMN-memory model achieves 61.6% in accuracy. This finding implies that inferring the matching features by multi-turns works better than single turn. Compared with the ESIM, our MIMN model increases 7.0% in accuracy. We further find that the performance of MIMN achieves 77.9% and 73.1% in accuracy of entailment and contradiction respectively, outperforming all previous models. From the accuracy distributions on N, E, and C in Table 4, we can see that the MIMN model is good at dealing with entailment and contradiction while achieves only average performance on neural.

Consequently, the experiment results show that our MIMN model achieves a new state-of-the-art performance on MPE test set.

All of our models perform well on the entailment label, which reveals that our models can aggregate information from multiple sentences for entailment judgment.

4.6 Experiments on SCITAIL

In this section, we study the effectiveness of our model on the SCITAIL dataset. Table 4 presents the results of our models and the previous models on this dataset. Apart from the results reported in the original paper [12]: Majority class, ngram, decomposable attention, ESIM and DGEM, we compare further with the current state-of-the-art model CAFE [32].

We can see that the MIMN model achieves 84.0% in accuracy on SCITAIL test set, which outperforms the CAFE by a margin of 0.5%. Moreover, the MIMN-gate+ReLU model exceeds the CAFE slightly. The MIMN model increases 13.3% in test accuracy compared with the ESIM, which again proves that multi-turn inference is better than one-pass inference.

5 Conclusion

In this paper, we propose the MIMN model for NLI task. Our model introduces a multi-turns inference mechanism to process multi-perspective matching features. Furthermore, the model employs the memory mechanism to carry proceeding inference information. In each turn, the inference is based on the current matching feature and previous memory. Experimental results on SNLI dataset show that the MIMN model is on par with the state-of-the-art models. Moreover, our model achieves new state-of-the-art results on the MPE and the SCITAL datasets. Experimental results prove that the MIMN model can extract important information from multiple premises for the final judgment. And the model is good at handling the relationships of entailment and contradiction.

Acknowledgements

This work is funded by Beijing Advanced Innovation for Language Resources of BLCU, the Fundamental Research Funds for the Central Universities in BLCU (No.17PT05) and Graduate Innovation Fund of BLCU (No.18YCX010).