A Compare-Aggregate Model with Latent Clustering for Answer Selection

TL;DR

This paper introduces a novel answer selection model that leverages pretrained language models and a latent clustering technique, achieving state-of-the-art results on benchmark datasets.

Contribution

It proposes a compare-aggregate model with latent clustering and transfer learning, improving answer selection performance in NLP tasks.

Findings

Achieved state-of-the-art performance on WikiQA and TREC-QA datasets.

Demonstrated the effectiveness of latent clustering in answer selection.

Showed benefits of transfer learning with pretrained language models.

Abstract

In this paper, we propose a novel method for a sentence-level answer-selection task that is a fundamental problem in natural language processing. First, we explore the effect of additional information by adopting a pretrained language model to compute the vector representation of the input text and by applying transfer learning from a large-scale corpus. Second, we enhance the compare-aggregate model by proposing a novel latent clustering method to compute additional information within the target corpus and by changing the objective function from listwise to pointwise. To evaluate the performance of the proposed approaches, experiments are performed with the WikiQA and TREC-QA datasets. The empirical results demonstrate the superiority of our proposed approach, which achieve state-of-the-art performance for both datasets.

Tables

Table 1. Table 1. Properties of the dataset.

Dataset	Listwise pairs			Pointwise pairs
	train	dev	test	train	dev	test
WikiQA	873	126	243	8.6k	1.1k	2.3k
TREC-QA	1.2k	65	68	53k	1.1k	1.4k
QNLI	86k	10k	-	428k	169k	-

Table 2. Table 2. Model performance (the top 3 scores are marked in bold for each task). We evaluate model  (Wang and Jiang, 2017 ; Bian et al . , 2017 ; Shen et al . , 2017 ; Tran et al . , 2018 ) on the WikiQA corpus using author’s implementation (marked by *). For TREC-QA case, we present reported results in the original papers.

Model	WikiQA				TREC-QA
	MAP		MRR		MAP		MRR
	dev	test	dev	test	dev	test	dev	test
Compare-Aggregate (2017) (Wang and Jiang, 2017)	0.743*	0.699*	0.754*	0.708*	-	-	-	-
Comp-Clip (2017) (Bian et al., 2017)	0.732*	0.718*	0.738*	0.732*	-	0.821	-	0.899
IWAN (2017) (Shen et al., 2017)	0.738*	0.692*	0.749*	0.705*	-	0.822	-	0.899
IWAN + sCARNN (2018) (Tran et al., 2018)	0.719*	0.716*	0.729*	0.722*	-	0.829	-	0.875
MCAN (2018) (Tay et al., 2018)	-	-	-	-	-	0.838	-	0.904
Question Classification (2018) (Madabushi et al., 2018)	-	-	-	-	-	0.865	-	0.904
Listwise Learning to Rank
Comp-Clip (our implementation)	0.756	0.708	0.766	0.725	0.750	0.744	0.805	0.791
Comp-Clip (our implementation) + LM	0.783	0.748	0.791	0.768	0.825	0.823	0.870	0.868
Comp-Clip (our implementation) + LM + LC	0.787	0.759	0.793	0.772	0.841	0.832	0.877	0.880
Comp-Clip (our implementation) + LM + LC +TL	0.822	0.830	0.836	0.841	0.866	0.848	0.911	0.902
Pointwise Learning to Rank
Comp-Clip (our implementation)	0.776	0.714	0.784	0.732	0.866	0.835	0.933	0.877
Comp-Clip (our implementation) + LM	0.785	0.746	0.789	0.762	0.872	0.850	0.930	0.898
Comp-Clip (our implementation) + LM + LC	0.782	0.764	0.785	0.784	0.879	0.868	0.942	0.928
Comp-Clip (our implementation) + LM + LC +TL	0.842	0.834	0.845	0.848	0.913	0.875	0.977	0.940

Table 3. Table 3. Model (Comp-Clip +LM +LC) performance on the QNLI corpus with a variant number of clusters (top score marked as bold).

# Clusters	Listwise Learning		Pointwise Learning
	MAP	MRR	MAP	MRR
1	0.822	0.819	0.842	0.841
4	0.839	0.840	0.846	0.845
8	0.841	0.842	0.846	0.846
16	0.840	0.842	0.847	0.846

Equations

$$\overline{\text{Q}}=\sigma ({\text{W}}^i\text{Q})\odot \text{tanh}({\text{W}}^u\text{Q}),$$

$$\overline{\text{A}}=\sigma ({\text{W}}^i\text{A})\odot \text{tanh}({\text{W}}^u\text{A}),$$

$${\text{H}}^Q=\overline{\text{Q}}\cdot \text{softmax}({({\text{W}}^q\overline{\text{Q}})}^{⊺}\overline{\text{A}}),$$

$${\text{H}}^A=\overline{\text{A}}\cdot \text{softmax}({({\text{W}}^a\overline{\text{A}})}^{⊺}\overline{\text{Q}}).$$

$${\text{C}}^Q=\overline{\text{A}}\odot {\text{H}}^Q,({\text{C}}^Q\in {\mathbb{R}}^{l\times A}),$$

$${\text{C}}^A=\overline{\text{Q}}\odot {\text{H}}^A,({\text{C}}^A\in {\mathbb{R}}^{l\times Q}),$$

$${\text{R}}^Q=\text{CNN}({\text{C}}^Q),{\text{R}}^A=\text{CNN}({\text{C}}^A),$$

$$\text{score}=\sigma ([{\text{R}}^Q;{\text{R}}^A{]}^{⊺}\text{W}),$$

$${\text{p}}_{1:n}={\text{s}}^{⊺}\text{W}{\text{M}}_{1:n},$$

$${\overline{\text{p}}}_{1:k}=\text{k-max-pool}({\text{p}}_{1:n}),$$

$${\alpha }_{1:k}=\text{softmax}({\overline{\text{p}}}_{1:k}),$$

$${\text{M}}_{\text{LC}}={\sum }_k{\overline{\alpha }}_k{\text{M}}_k,$$

$${\text{M}}_{\text{LC}}^Q=f(({\sum }_i{\overline{q}}_i)/n),{\overline{q}}_i\subset {\overline{\text{Q}}}_{1:n},$$

$${\text{M}}_{\text{LC}}^A=f(({\sum }_i{\overline{a}}_i)/m),{\overline{a}}_i\subset {\overline{\text{A}}}_{1:m},$$

$${\text{C}}_{\text{new}}^Q=[{\text{C}}^Q;{\text{M}}_{\text{LC}}^Q],{\text{C}}_{\text{new}}^A=[{\text{C}}^A;{\text{M}}_{\text{LC}}^A],$$

$${\text{score}}_i=\text{model}(\text{Q},{\text{A}}_i),$$

$$\text{S}=\text{softmax}([{\text{score}}_1,...,{\text{score}}_i]),$$

$$\text{loss}={\sum }_{n=1}^N\text{KL}({\text{S}}_n||{\text{y}}_n),$$

$$\text{loss}=-{\sum }_{n=1}^N{y}_n\text{log}({\text{score}}_n),$$

Full text

question answering; natural language processing; information retrieval; deep learning

††journalyear: 2019††copyright: acmcopyright††conference: The 28th ACM International Conference on Information and Knowledge Management; November 3–7, 2019; Beijing, China††booktitle: The 28th ACM International Conference on Information and Knowledge Management (CIKM’19), November 3–7, 2019, Beijing, China††price: 15.00††doi: 10.1145/3357384.3358148††isbn: 978-1-4503-6976-3/19/11

1. Introduction

Automatic question answering (QA) is a primary objective of artificial intelligence. Recently, research on this task has taken two major directions based on the answer span considered by the model. The first direction (i.e., the fine-grained approach) finds an exact answer to a question within a given passage (Rajpurkar et al., 2016). The second direction (i.e., the coarse-level approach) is an information retrieval (IR)-based approach that provides the most relevant sentence from a given document in response to a question. In this study, we are interested in building a model that computes a matching score between two text inputs. In particular, our model is designed to undertake an answer-selection task that chooses the sentence that is most relevant to the question from a list of answer candidates. This task has been extensively investigated by researchers because it is a fundamental task that can be applied to other QA-related tasks (Wang and Jiang, 2017; Bian et al., 2017; Shen et al., 2017; Tran et al., 2018; Tay et al., 2018; Madabushi et al., 2018).

However, most previous answer-selection studies have employed small datasets (Wang et al., 2007; Yang et al., 2015) compared with the large datasets employed for other natural language processing (NLP) tasks (Lowe et al., 2015; Rajpurkar et al., 2016). Therefore, the exploration of sophisticated deep learning models for this task is difficult.

To fill this gap, we conduct an intensive investigation with the following directions to obtain the best performance in the answer-selection task. First, we explore the effect of additional information by adopting a pretrained language model (LM) to compute the vector representation of the input text. Recent studies have shown that replacing the word-embedding layer with a pretrained language model helps the model capture the contextual meaning of words in the sentence (Peters et al., 2018; Devlin et al., 2019). Following this study, we select an ELMo (Peters et al., 2018) language model for this study. We investigate the applicability of transfer learning (TL) using a large-scale corpus that is created for a relevant-sentence-selection task (i.e., question-answering NLI (QNLI) dataset (Wang et al., 2018)). Second, we further enhance one of the baseline models, Comp-Clip (Bian et al., 2017) (refer to the discussion in 3.1), for the target QA task by proposing a novel latent clustering (LC) method. The LC method computes latent cluster information for target samples by creating a latent memory space and calculating the similarity between the sample and the memory. By an end-to-end learning process with the answer-selection task, the LC method assigns true-label question-answer pairs to similar clusters. In this manner, a model will have further information for matching sentence pairs, which increases the total model performance. Last, we explore the effect of different objective functions (listwise and pointwise learning). In contrast to previous research (Bian et al., 2017), we observe that the pointwise learning approach performs better than the listwise learning approach when we apply our proposed methods. Extensive experiments are conducted to investigate the efficacy and properties of the proposed methods and show the superiority of our proposed approaches for achieving state-of-the-art performance with the WikiQA and TREC-QA datasets.

2. Related Work

Researchers have investigated models based on neural networks for question-answering tasks. One study employs a Siamese architecture that utilizes an encoder (e.g., RNN or CNN) to compute vector representations of the question and the answer. The affinity score is calculated based on these vector representations (Lowe et al., 2015). To improve the model performance by enabling the use of information from one sentence (e.g., a question or an answer) in computing the representation of another sentence, researchers included the attention mechanism in their models (Tan et al., 2015; Santos et al., 2016; Wang et al., 2017).

Another line of research includes the compare-aggregate framework (Wang and Jiang, 2017). In this framework, first, vector representations of each sentence are computed. Second, these representations are compared. Last, the results are aggregated to calculate the matching score between the question and the answer (Bian et al., 2017; Shen et al., 2017; Tran et al., 2018).

In this study, unlike the previous research, we employ a pretrained language model and a latent-cluster method to help the model understand the information in the question and the answer.

3. Methods

3.1. Comp-Clip Model

In this paper, we are interested in estimating the matching score $f(y|\textbf{Q},\textbf{A})$, where $y$, $\textbf{Q}\,{=}\,\{q_{1},...,q_{n}\}$ and $\textbf{A}\,{=}\,\{a_{1},...,a_{m}\}$ represent the label, the question and the answer, respectfully. We select the model from (Bian et al., 2017), which is referred to as the Comp-Clip model, as our baseline model. The model consists of the following four parts:

**Context representation: ** The question $\textbf{Q}\,{\in}\,\mathbb{R}^{d\times Q}$ and answer $\textbf{A}\,{\in}\,\mathbb{R}^{d\times A}$, (where $d$ is a dimensionality of word embedding and Q and A are the length of the sequence in Q and A, respectively), are processed to capture the contextual information and the word as follows:

[TABLE]

where $\odot$ denotes element-wise multiplication, and $\sigma$ is the sigmoid function. The $\textbf{W}\,{\in}\,\mathbb{R}^{l\times d}$ is the learned model parameter.

**Attention: ** The soft alignment of each element in $\overline{\textbf{Q}}\,{\in}\,\mathbb{R}^{l\times Q}$ and $\overline{\textbf{A}}\,{\in}\,\mathbb{R}^{l\times A}$ are calculated using dynamic-clip attention (Bian et al., 2017). We obtain the corresponding vectors $\textbf{H}^{Q}\,{\in}\,\mathbb{R}^{l\times A}$ and $\textbf{H}^{A}\,{\in}\,\mathbb{R}^{l\times Q}$.

[TABLE]

**Comparison: ** A comparison function is used to match each word in the question and answer to a corresponding attention-applied vector representation:

[TABLE]

where $\odot$ denotes element-wise multiplication.

**Aggregation: ** We aggregate the vectors from the comparison layer using CNN (Kim, 2014) with $n$-types of filters and calculate the matching score between Q and A.

[TABLE]

where [;] denotes concatenation of each vector $\textbf{R}^{Q}\,{\in}\,\mathbb{R}^{nl}$ and $\textbf{R}^{A}\,{\in}\,\mathbb{R}^{nl}$. The $\textbf{W}\,{\in}\,\mathbb{R}^{2nl\times 1}$ is the learned model parameter.

3.2. Proposed Approaches

To achieve the best performance in the answer-selection task, we propose four approaches: adding a pretrained LM; adding the LC information of each sentence as auxiliary knowledge; applying TL to benefit from large-scale data; and modifying the objective function from listwise to pointwise learning. Figure 1 depicts the total architecture of the proposed model.

**Pretrained Language Model (LM): ** Recent studies have shown that replacing the word embedding layer with a pretrained LM helps the model capture the contextual meaning of the words in the sentence (Peters et al., 2018; Devlin et al., 2019). We select an ELMo (Peters et al., 2018) language model and replace the previous word embedding layer with the ELMo model as follows: $\textbf{L}^{\text{Q}}\,{=}\,\text{ELMo}(\textbf{Q}),~{}\textbf{L}^{\text{A}}\,{=}\,\text{ELMo}(\textbf{A})$. These new representations—$\textbf{L}^{\text{Q}}$ and $\textbf{L}^{\text{A}}$—are substituted for Q and A, respectively, in equation (1).

**Latent Clustering (LC) Method: ** We assume that extracting the LC information of the text and using it as auxiliary information will help the neural network model analyze the corpus. The dotted box in figure 1 shows the proposed LC method. We create n-many latent memory vectors $\textbf{M}_{1:n}$ and calculate the similarity between the sentence representation and each latent memory vector. The latent-cluster information of the sentence representation will be obtained using a weighted sum of the latent memory vectors according to the calculated similarity as follows:

[TABLE]

where $\textbf{s}\,{\in}\,\mathbb{R}^{d}$ is a sentence representation, $\textbf{M}_{1:n}\,{\in}\,\mathbb{R}^{d^{\prime}\times n}$ indicates the latent memory, and $\textbf{W}{\in}\,\mathbb{R}^{d\times d^{\prime}}$ is the learned model parameter.

We apply the LC method and extract cluster information from each question and answer. This additional information is added to each of the final representations in the comparison part (see 3.1) as follows:

[TABLE]

where $f$ is the LC method (in equation 5) and [;] denotes the concatenation of each vector. These new representations—$\textbf{C}^{Q}_{\text{new}}$ and $\textbf{C}^{A}_{\text{new}}$—are substituted for $\textbf{C}^{Q}$ and $\textbf{C}^{A}$ in equation (4). Note that we average word-embedding to obtain sentence representation in the previous equation.

**Transfer Learning (TL): ** To observe the efficacy in a large dataset, we apply transfer learning using the question-answering NLI (QNLI) corpus (Wang et al., 2018). We train the CompClip model with the QNLI corpus and then fine-tune the model with target corpora, such as the WikiQA and TREC-QA datasets.

**Pointwise Learning to Rank: ** Previous research adopts a listwise learning approach. With a dataset that consists of a question, Q, a related answer set, $\textbf{A}\,{=}\,\{\textbf{A}_{1},...,\textbf{A}_{N}\}$, and a target label, $\textbf{y}\,{=}\,\{y_{1},...,y_{N}\}$, a matching score is computed using equation (4). This approach applies KL-divergence loss to train the model as follows:

[TABLE]

where $i$ is the number of answer candidates for the given question and $N$ is the total number of samples employed during training.

In contrast, we pair each answer candidate to the question and compute the cross-entropy loss to train the model as follows:

[TABLE]

where $N$ is the total number of samples used during training. Using this approach, the number of training instances for a single iteration increases, as shown in table 1.

4. Empirical Results

We regard all tasks as relevant answer selections for the given questions. Following the previous study, we report the model performance as the mean average precision (MAP) and the mean reciprocal rank (MRR)111https://aclweb.org/aclwiki/Question_Answering_(State_of_the_art). To test the performance of the model, we utilize the TREC-QA, WikiQA and QNLI datasets (Wang et al., 2007; Yang et al., 2015; Wang et al., 2018).

4.1. Dataset

WikiQA (Yang et al., 2015) is an answer selection QA dataset constructed from real queries of Bing and Wikipedia. Following the literature (Bian et al., 2017; Shen et al., 2017), we use only questions that contain at least one correct answer among the list of answer candidates. There are 873/126/243 questions and 8,627/1,130/2,351 question-answer pairs for train/dev/test split.

TREC-QA (Wang et al., 2007) is another answer selection QA dataset created from the TREC Question-Answering tracks. In this study, we use the clean dataset that removed questions from the dev and test datasets that did not have answers or had only positive/negative answers. There are 1,229/65/68 questions and 53,417/1,117/1,442 question-answer pairs for train/dev/test split.

QNLI (Wang et al., 2018) is a modified version of the SQuAD dataset (Rajpurkar et al., 2016) that allows for sentence selection QA. The context paragraph in SQuAD is split into sentences, and each sentence is paired with the question. The true label is given to the question-sentence pairs when the sentence contains the answer. There are 86,308/10,385 questions and 428,998/169,435 question-answer pairs for train/dev split. Considering the large size of this dataset, we use it to train the base model for transfer learning; it is also used to evaluate the proposed model performance in a large dataset environment.

4.2. Implementation Details

To implement the Comp-Clip model, we apply a context projection weight matrix with 100 dimensions that are shared between the question part and the answer part (eq. 1). In the aggregation part, we use 1-D CNN with a total of 500 filters, which involves five types of filters $K\,{\in}\,\mathbb{R}^{\{1,2,3,4,5\}\times 100}$, 100 per type. This CNN is independently applied to the question part and answer part. For the LC method, we perform additional hyper-parameter searching experiments to select the best parameters. We select k (for the k-max-pool in equation 5) as 6 and 4 for the WikiQA and TREC-QA case, respectively. In both datasets, we apply 8 latent clusters.

The vocabulary size in the WiKiQA, TREC-QA and QNLI dataset are 30,104, 56,908 and 154,442, respectively. When applying the TL, the vocabulary size is set to 154,442, and the dimension of the context projection weight matrix is set to 300. We use the Adam optimizer, including gradient clipping, by the norm at a threshold of 5. For the purpose of regularization, we applied a dropout with a ratio of 0.5.

4.3. Comparison with Other Methods

Table 2 shows the model performance for the WikiQA and TREC-QA datasets. For the Compare-Aggregate (2016), Comp-Clip (2017), IWAN (2017) and IWAN+sCARNN (2018) models, we measure the performance on the WikiQA dataset using the authors’ implementations (marked by * in the table). Unlike previous studies, we report our results for both the dev dataset and the test dataset because we note a performance gap between these datasets. While training the model, we apply an early stop that is based on the performance of the dev dataset and measure the performance on the test dataset. Because Comp-Clip (Bian et al., 2017) is our baseline model, we implement it from scratch and achieve a performance that is similar to that of the original paper.

**WikiQA: ** For the WikiQA dataset, the pointwise learning approach shows a better performance than the listwise learning approach. We combine LM with the base model (Comp-Clip +LM) and observe a significant improvement in performance in terms of MAP (0.714 to 0.746 absolute). When we add the LC method (Comp-Clip +LM +LC), the best previous results are surpassed in terms of MAP (0.718 to 0.764 absolute). We achieve a vast improvement in performance in terms of the MAP (0.764 to 0.834 absolute) by including the TL approach (Comp-Clip + LM + LC + TL).

**TREC-QA: ** The pointwise learning approach also shows excellent performance with the TREC-QA dataset. As shown in table 1, the TREC-QA dataset has a larger number of answer candidates per question. We assume that this characteristic prevents the model from handling the dataset with a listwise learning approach. As in the WikiQA case, we achieve additional performance gains in terms of the MAP as we apply LM, LC, and TL (0.850, 0.868 and 0.875, respectively). In particular, our model outperforms the best previous result when we add LC method, (Comp-Clip +LM +LC) in terms of MAP (0.865 to 0.868).

4.4. Impact of Latent Clustering

To evaluate the impact of latent clustering method (Comp-Clip +LM +LC) in a larger dataset environment, we perform QNLI evaluation. Table 3 shows the performance of the model (Comp-Clip +LM +LC) for the QNLI dataset with a variant number of clusters. Note that the QNLI dataset is created from the SQuAD (Rajpurkar et al., 2016) dataset, which only provides train and dev subsets. Consequently, we report the model performances for the dev dataset. As shown in the table, we achieve the best results with 8 clusters in listwise learning and 16 clusters in pointwise learning. In both cases, we achieve no additional performance gain after 16 clusters.

5. Conclusion

In this study, our proposed method achieves state-of-the-art performance for both the WikiQA dataset and TREC-QA dataset. We show that leveraging a large amount of data is crucial for capturing the contextual representation of input text. In addition, we show that the proposed latent clustering method with a pointwise objective function significantly improves the model performance in the sentence-level QA task.

Acknowledgements.

We sincerely thank Carl I. Dockhorn and Yu Gong at Adobe for their in-depth feedback for this research. K. Jung is with ASRI, Seoul National University, Korea. This work was supported by the Ministry of Trade, Industry & Energy (MOTIE, Korea) under Industrial Technology Innovation Program (No.10073144) and by the NRF funded by the Korea government (MSIT) (No. 2016M3C4A7952632).