Machine Reading Comprehension for Answer Re-Ranking in Customer Support   Chatbots

Momchil Hardalov; Ivan Koychev; Preslav Nakov

arXiv:1902.04574·cs.CL·February 27, 2019

Machine Reading Comprehension for Answer Re-Ranking in Customer Support Chatbots

Momchil Hardalov, Ivan Koychev, Preslav Nakov

PDF

TL;DR

This paper introduces a machine reading comprehension-based re-ranking method for customer support chatbots, improving answer relevance by leveraging deep neural models trained on Twitter customer support data.

Contribution

It adapts machine reading comprehension techniques for answer re-ranking in customer support chatbots, enhancing response quality over traditional methods.

Findings

01

Improved answer relevance in chatbot responses.

02

Enhanced semantic matching between questions and answers.

03

Effective re-ranking using neural models trained on real-world data.

Abstract

Recent advances in deep neural networks, language modeling and language generation have introduced new ideas to the field of conversational agents. As a result, deep neural models such as sequence-to-sequence, Memory Networks, and the Transformer have become key ingredients of state-of-the-art dialog systems. While those models are able to generate meaningful responses even in unseen situation, they need a lot of training data to build a reliable model. Thus, most real-world systems stuck to traditional approaches based on information retrieval and even hand-crafted rules, due to their robustness and effectiveness, especially for narrow-focused conversations. Here, we present a method that adapts a deep neural architecture from the domain of machine reading comprehension to re-rank the suggested answers from different models using the question as context. We train our model using…

Figures5

Click any figure to enlarge with its caption.

Figure 1

Figure 2

Tables4

Table 1. Table 1: Statistics about our dataset. (Reprinted by permission from Springer Nature: Springer Lecture Notes in Computer Science (Hardalov, M.; Koychev, I.; Nakov, P. Towards Automated Customer Support Hardalov et al. ( 2018 ) ), 2018)

	Questions	Answers
Avg. # words	$21.314 814$	$25.884 436$
Min # words	$1.000 000$	$3.000 000$
1st quantile (#words)	$13.000 000$	$20.000 000$
Mode (# words)	$20.000 000$	$23.000 000$
3rd quantile (#words)	$27.000 000$	$29.000 000$
Max # words	$136.000 000$	$70.000 000$
Overall
# question–answer pairs		49,626
# words (in total)		26,140
Min # turns per dialog		$2$
Max # turns per dialog		$106$
Avg. # turns per dialog		2.6
Training set: # of dialogs		45,582
Testing set: # of dialogs		4,044

Table 2. Table 2: Auxiliary task: question–answer goodness classification results.

Model	Embedding Type	d_model	Heads	Accuracy
Majority class	–	–	–	$50.515 625$
QANet	GloVe	64	4	$80.578 125$
		64	8	$82.828 12$
		128	8	$83.421 874$
QANet	ELMo (token level)	64	4	$82.921 875$
		64	8	$83.875$
		128	8	$83.484 375$
QANet	ELMo (sentence level)	64	8	$84.093 75$
QANet	ELMo (sentence level)	128	8	$85.453 12$

Table 3. Table 3: Main task: performance of the individual models.

	Word Overlap		Semantic Similarity
Model	BLEU@2	ROUGE_L	Emb Avg	Greedy Match	Vec Extr
Transformer Hardalov et al. (2018)	$12.427 277 110 577 412$	$25.328 589 592 296 84$	$75.350 357 724 641 75$	$30.077 073 951 984 16$	$39.398 955 533 090 98$
IR-BM25 Hardalov et al. (2018)	$13.732 301 176 466 532$	$22.347 860 605 005 877$	$76.526 586 397 381 92$	$29.719 912 622 416 917$	$37.986 670 995 115 75$
seq2seq Hardalov et al. (2018)	$15.104 511 595 780 986$	$26.597 981 802 417 063$	$77.109 255 163 930 53$	$30.809 532 318 667 042$	$40.231 592 168 970 955$
QANet on IR (Individual)	$14.918 819 893 \pm 0.128 209 477$	$23.299 038 571 \pm 0.121 935 742$	$77.470 088 663 \pm 0.059 103 246$	$30.396 535 191 \pm 0.059 016 826$	$39.627 030 115 \pm 0.056 555 891$

Table 4. Table 4: Main task: re-ranking the top K = 5 𝐾 5 K=5 answers returned by the IR and the seq2seq models.

Model	Word Overlap		Semantic Similarity
Model	BLEU@2	ROUGE_L	Emb Avg	Greedy Match	Vec Extr
Random Top Answer	$14.516 328 934 \pm 0.123 082 790$	$23.413 930 960 \pm 0.118 796 211$	$77.214 657 500 \pm 0.064 671 076$	$30.244 576 358 \pm 0.065 005 588$	$38.247 697 434 \pm 0.199 359 940$
QANet+GloVe
d=64, h=4	$15.178 345 469 681 423$	$24.127 489 239 551 984$	$78.383 814 994 509 07$	$31.142 418 111 378 483$	$40.854 223 779 457 804$
Softmax	$15.812 672 681 \pm 0.089 978 745$	$24.527 707 352 \pm 0.053 996 400$	$78.315 529 844 \pm 0.079 793 894$	$31.097 286 872 \pm 0.028 871 124$	$40.509 609 850 \pm 0.122 710 191$
d=64, h=8	$15.413 995 490 859 303$	$23.615 145 621 599 66$	$78.483 518 493 617 24$	$30.969 496 032 273 93$	$40.805 487 519 474 156$
Softmax	$15.897 469 640 \pm 0.060 322 919$	$24.392 476 297 \pm 0.028 498 873$	$78.381 206 911 \pm 0.038 577 508$	$31.106 734 601 \pm 0.017 817 047$	$40.662 627 653 \pm 0.056 110 335$
d = 128, h = 8	$15.940 853 825 830 132$	$24.589 339 773 822 14$	$78.291 357 307 508 48$	$31.191 942 261 508 515$	$40.626 509 935 461 46$
Softmax	$16.035 070 659 \pm 0.080 839 093$	$24.714 711 934 \pm 0.063 554 276$	$78.362 449 621 \pm 0.074 696 344$	$31.196 202 485 \pm 0.067 497 757$	$40.704 445 016 \pm 0.053 693 214$
QANet+ELMo (Token)
d = 64, h = 4	$15.230 794 657 823 363$	$23.482 390 989 589 16$	$78.246 333 058 784$	$30.769 323 099 246 908$	$40.217 083 745 604 9$
Softmax	$15.774 599 918 \pm 0.145 520 634$	$24.441 624 727 \pm 0.092 950 497$	$78.272 501 118 \pm 0.032 568 643$	$31.062 447 538 \pm 0.048 212 825$	$40.459 904 884 \pm 0.105 238 931$
d = 64, h = 8	$15.302 739 194 160 967$	$23.407 297 463 506 286$	$78.542 117 611 342 08$	$30.966 868 253 418 33$	$40.189 469 793 465 18$
Softmax	$15.858 178 111 \pm 0.068 446 490$	$24.397 360 279 \pm 0.064 433 766$	$78.360 976 244 \pm 0.075 414 933$	$31.111 994 054 \pm 0.037 515 481$	$40.491 637 800 \pm 0.050 358 916$
d = 128, h = 8	$15.238 346 336 693 425$	$23.586 940 133 301 03$	$78.341 047 379 888 61$	$30.895 847 682 105 25$	$40.187 819 524 983 74$
Softmax	$15.890 561 845 \pm 0.075 167 358$	$24.552 827 022 \pm 0.100 807 807$	$78.328 864 540 \pm 0.058 647 350$	$31.110 185 304 \pm 0.047 704 418$	$40.400 958 397 \pm 0.053 828 640$
QANet+ELMo (Sentence)
d = 64, h = 8	$15.476 033 595 794 37$	$23.881 348 518 540 29$	$78.435 553 876 916 27$	$30.961 283 999 884 152$	$40.332 224 328 956 094$
Softmax	$15.995 253 684 \pm 0.142 636 417$	$24.499 412 456 \pm 0.330 357 113$	$78.341 333 969 \pm 0.097 839 071$	$31.129 990 155 \pm 0.083 475 562$	$40.562 566 876 \pm 0.087 607 130$
d = 128, h = 8	$15.643 673 497 249 111$	$24.132 396 421 511 17$	$78.516 467 311 058 34$	$31.139 380 135 359 822$	$40.632 245 181 311 14$
Softmax	$16.054 222 562 \pm 0.056 937 106$	$24.807 205 017 \pm 0.084 232 102$	$78.398 183 357 \pm 0.070 499 616$	$31.201 663 932 \pm 0.056 794 468$	$40.579 787 762 \pm 0.029 219 118$

Equations12

A n s = a \in A arg max (o (q, a))

A n s = a \in A arg max (o (q, a))

p ∣ q, A \sim softmax (o (q, a_{1}), \dots, o (q, a_{K}))

p ∣ q, A \sim softmax (o (q, a_{1}), \dots, o (q, a_{K}))

A n s ∣ p \sim Cat (K, p)

A n s ∣ p \sim Cat (K, p)

g r ee d y (u_{1}, u_{2}) = \frac{\sum _{v \in u_{1}} w e i g h t ( v ) * max _{w \in u_{2}} cos ( v , w )}{\sum _{v \in u_{1}} w e i g h t ( v )}

g r ee d y (u_{1}, u_{2}) = \frac{\sum _{v \in u_{1}} w e i g h t ( v ) * max _{w \in u_{2}} cos ( v , w )}{\sum _{v \in u_{1}} w e i g h t ( v )}

s im G r ee d y (u_{1}, u_{2}) = \frac{g r ee d y ( u _{1} , u _{2} ) + g r ee d y ( u _{2} , u _{1} )}{2}

s im G r ee d y (u_{1}, u_{2}) = \frac{g r ee d y ( u _{1} , u _{2} ) + g r ee d y ( u _{2} , u _{1} )}{2}

e x t r e ma (u_{i}) = {max u_{i}, min u_{i}, i f max u_{i} \geq ∣ min u_{i} ∣ otherwise

e x t r e ma (u_{i}) = {max u_{i}, min u_{i}, i f max u_{i} \geq ∣ min u_{i} ∣ otherwise

Peer Reviews

No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.

Videos

No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.

Taxonomy

MethodsLinear Layer · Absolute Position Encodings · Position-Wise Feed-Forward Layer · Residual Connection · Byte Pair Encoding · Dense Connections · Label Smoothing · *Communicated@Fast*How Do I Communicate to Expedia? · Adam · Softmax

Full text

Abstract

Recent advances in deep neural networks, language modeling and language generation have introduced new ideas to the field of conversational agents. As a result, deep neural models such as sequence-to-sequence, memory networks, and the Transformer have become key ingredients of state-of-the-art dialog systems. While those models are able to generate meaningful responses even in unseen situations, they need a lot of training data to build a reliable model. Thus, most real-world systems have used traditional approaches based on information retrieval (IR) and even hand-crafted rules, due to their robustness and effectiveness, especially for narrow-focused conversations. Here, we present a method that adapts a deep neural architecture from the domain of machine reading comprehension to re-rank the suggested answers from different models using the question as a context. We train our model using negative sampling based on question–answer pairs from the Twitter Customer Support Dataset. The experimental results show that our re-ranking framework can improve the performance in terms of word overlap and semantics both for individual models as well as for model combinations.

keywords:

conversational agents; chatbots; machine reading comprehension; question answering; information retrieval; answer re-ranking.

\pubvolume

10 \issuenum1 \articlenumber82

\historyReceived: date; Accepted: date; Published: date

\TitleMachine Reading Comprehension for Answer Re-Ranking in Customer Support Chatbots \AuthorMomchil Hardalov 1,∗, Ivan Koychev 1 and Preslav Nakov 2 \AuthorNamesMomchil Hardalov, Ivan Koychev and Preslav Nakov

\corresCorrespondence: [email protected]

1 Introduction

The growing popularity of smart devices, personal assistants, and online customer support systems has driven the research community to develop various new methodologies for automatic question answering and chatbots. In the domain of conversational agents, two general types of systems have become dominant: (i) retrieval-based, and (ii) generative. While the former produce clear and smooth output, the latter bring flexibility and the ability to generate new unseen answers.

In this work, we focus on finding the most suitable answer for a question, where each candidate can be produced by a different system, e.g., knowledge-based, rule-based, deep neural network, retrieval, etc. In particular, we propose a re-ranking framework based on machine reading comprehension Yu et al. (2018); Seo et al. (2017); Chen et al. (2017) for question–answer pairs. Moreover, instead of selecting the top candidate from the re-ranker’s output, we use probabilistic sampling that aims to diversify the agent’s language and to up-vote popular answers from different input models. We train our model using negative sampling based on question–answer pairs from the Twitter Customer Support Dataset.

In our experimental setup, we adopt a real-world application scenario, where we train on historical logs for some period of time, and then we test on logs for subsequent days. We evaluate the model using both semantic similarity measures, as well as word-overlap ones such as BLEU Papineni et al. (2002) and ROUGE Lin (2004), which come from machine translation and text summarization.

The remainder of this paper is organized as follows: Section 2 presents some related work in the domain of conversational agents and answer combination. Section 3 describes our framework and the general workflow for answer re-ranking. Section 4 introduces the original dataset and explains how we used in to build a new, task-specific one with negative sampling; it also offers insights about the dialogs and the pre-processing. Section 5 describes our experiments, and gives details about the training parameters. Section 6 presents the performance of each model and discusses the results. Finally, Section 7 concludes and suggests possible directions for future work.

2 Related Work

2.1 Conversational Agents

The emergence of large conversational corpora such as the Ubuntu Dialog corpus (Lowe et al., 2015), OpenSubtitles (Lison and Tiedemann, 2016), CoQA (Reddy et al., 2018) and the Microsoft Research Social Media Conversation Corpus (Microsoft Research, 2014) has enabled the use of generative models and end-to-end neural networks in the domain of conversational agents. In particular, sequence-to-sequence (seq2seq) models, which were initially proposed for machine translation (Luong et al., 2015; Sutskever et al., 2014; Bahdanau et al., 2014), got adapted to become a standard tool for training end-to-end dialogue systems. Early vanilla seq2seq models (Vinyals and Le, 2015) got quickly extended to model hierarchical structure (Serban et al., 2015), context (Sordoni et al., 2015a), and combination thereof (Sordoni et al., 2015b). While models were typically trained on corpora such as Ubuntu, some work (Boyanov et al., 2017) has also used data from Community Question Answering forums (Nakov et al., 2017); this means forming a training pair involving a question and each good answer in the corresponding question-answer thread.

More recently, the Transformer, a model without recurrent connections, was proposed (Vaswani et al., 2017), demonstrating state-of-the-art results for Machine Translation in various experimental scenarios for several language pairs and translation directions, thus, emerging as a strong alternative to seq2seq methods. The fact that it only uses self-attention makes it a lot faster both at training and at inference time, even though its deep architecture requires more calculations than a seq2seq model, it enables high degree of parallelism, while maintaining the ability to model word sequences through the mechanism of attention and positional embeddings.

In the domain of customer support, it has been shown that generative models such as seq2seq and the Transformer perform better then retrieval-based models, but they fail in the case of insufficient training data (Hardalov et al., 2018). Other works have incorporated intent categories and semantic matching into an answer selection model, which uses a knowledge base as its source (Li et al., 2018). In the insurance domain, Feng et al. (2015) proposed a generic deep learning approach for answer selection, based on convolutional neural networks (CNN) LeCun et al. (1989). In Li et al. (2015) combined recurrent neural networks based on long short-term memory (LSTM) cells Hochreiter and Schmidhuber (1997) and reinforcement learning (RL) to learn without the need of prior domain knowledge.

2.2 Answer Combination

Answer combination has been recognized as an important research direction in the domain of customer support chatbots. For example, Qiu et al. (2017) used an attentive seq2seq re-ranker to choose dynamically between the outputs of a retrieval-based and a seq2seq model. Similarly, Cui et al. (2017) combined a fact database, FAQs, opinion-oriented answers, and a neural-based chit-chat generator, by training a meta-engine that chooses between them.

Answer combination is also a key research topic in the related field of information retrieval (IR). For example, Pang et al. (2017) proposed a generic relevance ranker based on deep learning and CNNs LeCun et al. (1989), which tries to maintain standard IR search engine characteristics, such as exact matching and query term importance, while enriching the results based on semantics, proximity heuristics, and diversification.

3 Re-Ranking Model

Our re-ranking framework uses a classifier based on QANet Yu et al. (2018), a state-of-the-art architecture for machine reading comprehension, to evaluate whether a given answer is a good fit for the target question. It then uses the posterior probabilities of the classifier to re-rank the candidate answers, as shown in Figure 1.

3.1 Negative Sampling

Our goal is to distinguish “good” vs. “bad” answers, but the original dataset only contains valid, i.e., “good” question–answer pairs. Thus, we use negative sampling Mikolov et al. (2013), where we replace the original answer to the target question with a random answer from the training dataset. We further compare the word-based cosine similarity between the original and the sampled answer, and, in some rare cases, we turn a “bad” answer into “good” one if it is too similar to the original “good” answer.

3.2 QANet Architecture

Machine reading comprehension aims to answer a question by looking to extract a string from a given text context. Here, we use that model to measure the goodness of a given question–answer pair.

The first layer of the network is a standard an embedding layer, which transforms words into low-dimensional dense vectors. Afterwards, a two-layer highway network (Srivastava et al., 2015) is added on top of the embedding representations. This allows the network to regulate the information flow using a gated mechanism. The output of this layer is of dimensionality $\#words\times d$ , where $\#words$ is the number of words in the encoded sentence (Note that it differs for the question vs. the answer. See Section 5.1 for more detail.) and $d$ is the input/output dimensionality of the model for all Transformer layers, which is required by the architecture.

We experiment with two types of input embeddings. First, we use 200-dimensional GloVe (Pennington et al., 2014) vectors trained on 27 billion Twitter posts. We compare their performance to ELMo Peters et al. (2018), a recently proposed way to train contextualized word representations. In ELMo, these word vectors are learned activation functions of the internal states of a deep bi-directional language model. The latter is built upon a single (embedding) layer, followed by two LSTM (Hochreiter and Schmidhuber, 1997) layers, which are fed the words from a target sentence in a forward and a backward direction, accordingly. We obtain the final embedding by taking a weighted average over all three layers as suggested in (Peters et al., 2018).

The embedding encoder layer is based on a convolution, followed by self-attention Vaswani et al. (2017) and a feed-forward network. We use a kernel size of seven, $d$ filters, and four convolutional layers within a block. The output of the layer is $f(layernorm(x))+x$ , where $layernorm$ is the layer normalization operation (Ba et al., 2016). The output again is mapped to $\#words\times d$ by a 1D convolution. The input and the embedding layers are learned separately for the question and the answer.

The attention layer is a standard module for machine reading comprehension models. We call it answer-to-question (A2Q) and question-to-answer (Q2A) attention, which are also known as context-query and query-context, respectively. Let us denote the output of the encoder for the question as $Q$ and for the answer as $A$ . In order to obtain the attention, the model first computes a matrix $S$ with similarities between each two words for the question and the answer, then the values are normalized using softmax. The similarity function is defined as follows: $f(a,q)=W_{0}[a;q;a\odot q]$ .

We adopt the notation $\overline{S}=softmax(S)$ , which is a softmax normalization over the rows of $S$ , and $\overline{\overline{S}}=softmax(S^{\intercal})$ is a normalization over the columns. Then, the two attention matrices are computed as $A2Q=\overline{S}\cdot Q^{\intercal}$ , and $Q2A=\overline{S}\cdot\overline{\overline{S}}^{\intercal}\cdot C^{\intercal}$ .

The attention layer is followed by a model layer, which takes as input the concatenation of $[a;a2q;a\odot a2q;a\odot q2a]$ , where we use small letters to denote rows from the original matrices. For the output layer, we learn two different representations by passing the output of the model layer to two residual blocks, applying dropout (Srivastava et al., 2014) only to the inputs of the first one. We predict the output as $P(a|q)=\sigma(W_{o}[M_{0};M_{1}])$ . The weights are learned by minimizing a binary cross-entropy loss.

3.3 Answer Selection

We experimented with two answer selection strategies: (i) max, and (ii) proportional sampling after softmax normalization. The former strategy is standard and it selects the answer with the highest score, while the latter one returns a random answer with probability proportional to the score returned by the softmax, aiming at increasing the variability of the answers.

For both strategies, we use a linear projection applied on the output of the last residual model block, which is shows as “linear block” in Figure 1. We can generalize the latter as follows: $o(q,a_{k})=W_{o}[M]$ , where $M$ is the concatenation of the outputs of one or more residual model blocks.

We present the formulation of the two strategies, as we introduce the following notation: $Ans$ is the selected utterance by the agent; $o(q,a_{k})$ is the output of the model before applying the sigmoid function; $q$ is the original question by the user; $A$ is the set of possible answers that we want to re-rank. Equation (1) shows the selection process in the max case.

[TABLE]

We empirically found that the answer selection based on the max strategy does not always perform well. As our experimental results in Tables 3 and 4 show, we can gain notable improvement by using proportional sampling after softmax normalization, instead of always selecting the answer with the highest probability. In our experiments, we model $Ans$ as a random variable that follows a categorical distribution over $K=|A|$ events (candidate answers). For each of the question–answer pairs ( $q$ , $a$ ), we define the probability $p$ that $a$ is a good answer to $q$ using softmax as shown in Equations (2) and (3). Finally, we draw a random sample from Equation (3) to obtain the best matching answer.

[TABLE]

4 Data

The data and the resources that could be used to train customer support conversational agents are generally very scarce, as companies keep conversations locked on their own proprietary support systems. This is due to customer privacy concerns and to companies not wanting to make public their know-how and the common issues about their products and services. An extensive 2015 survey on available dialog corpora by Serban et al. (2018) found no good publicly available dataset for real-world customer support.

In early 2018, this situation changed as a new open dataset for Customer Support on Twitter (Twitter, 2019) was made available on Kaggle. It contains 3M tweets and replies for twenty big companies such as Amazon, Apple, Uber, Delta, and Spotify, among others. As customer support topics from different organizations are generally unrelated to each other, we focus only on tweets related to Apple support, which represents the largest number of tweets in the corpus.

We filtered all utterances that redirect the user to another communication channel, e.g., direct messages, which are not informative for the model and only bring noise. Moreover, since answers evolve over time, we divided our dataset into a training and a testing part, keeping earlier posts for training and the latest ones for testing. We further excluded from the training set all conversations that are older then sixty days. For evaluation, we used dialogs from the last five days in the dataset, to simulate a real-world scenario for customer support. We ended up with a dataset of 49,626 question–answer pairs divided into 45,582 for training and 4,044 for testing. Finally, we open-sourced our code for pre-processing and filtering the data, making it available to the research community (customer-support-chatbot, 2018).

Table 1 shows some statistics about our dataset. On the top of the table, we can see that the average number of turns per dialog is under three, which means that most of the dialogues finish after one answer from the customer support. The bottom of the table shows the distribution of the words in the user questions vs. the customer support answers. We can see that answers tend to be slightly longer, which is natural as replies by customer support must be extensive and helpful.

5 Experiments

5.1 Preprocessing

Since Twitter has its own specifics of writing in terms of both length (by design, tweets have been strictly limited to 140 characters; this constraint has been relaxed to 280 characters in 2017) and style, standard text tokenization is generally not suitable for tweets. Therefore, we used a specialized Twitter tokenizer Manning et al. (2014) to preprocess the data. Then, we further replaced shorthand entries such as ’ll, ’d, ’re, ’ve, with the most corresponding literary form, e.g., will, would, are, have. We also replaced shortened slang words, e.g., ’bout and ’til, with the standard words, e.g., about and until. Similarly, we replaced URLs with the special word <url>, all user mentions with <user>, and all hashtags with <hashtag>.

Due to the nature of writing in Twitter and the free form of the conversation, some of the utterances contain emoticons and emojis. They are handled automatically by the Twitter tokenizer and treated as a single token. We keep them in their original form, as they can be very useful for detecting emotions and sarcasm, which pose serious challenges for natural language understanding.

Based on the statistics presented in Section 4, we chose to trim the length of the questions and of the answers to 60 and 70 words, respectively.

5.2 Training Setup

For training, we use the Adam Kingma and Ba (2015) optimizer with decaying learning rate, as implemented in TensorFlow Abadi et al. (2016). We start with the following values: learning rate $\eta=$ 5\text{\times}{10}^{-04} $, exponential decay rate for the 1st and the 2nd momentum $\beta_{1}=$0.9$ and $\beta_{2}=$ 0.999 $, and constant for prevention of division by zero $\epsilon=$1\text{\times}{10}^{-7}$ . Then, we decay the learning after each epoch by a factor of 0.99. We also apply dropout with a probability of 0.1, and L2 weight decay on all trainable variables with $\lambda=$ 3\text{\times}{10}^{-7}$$. We train each model for 42K steps with a batch size of 64. We found these values by running a grid search on a dev set (extracted as a fraction of the training data) and using the values suggested in (Yu et al., 2018), where applicable.

5.3 Individual Models

Following (Hardalov et al., 2018), we experiment with three individual models: (i) Information Retrieval-based (IR), (ii) Sequence-to-sequence (seq2seq) and (iii) the Transformer.

For IR, we use ElasticSearch (ElasticSearch, 2019) with English analyzer enabled, whitespace- and punctuation-based tokenization, and word 3-grams. We further use the default BM25 algorithm Robertson and Zaragoza (2009), which is an improved version of TF.IDF. For all training questions and for all testing queries, we append the previous turns in the dialog as context.

For seq2seq, we use a bi-directional LSTM network with 512 hidden units per direction. The decoder has two uni-directional layers connected directly to the bi-directional layer in the encoder. The network takes as input words encoded as 200-dimensional embeddings. It is a combination of pre-trained GloVe Pennington et al. (2014) vectors for the known words, and a positional embedding layer, learned as model parameters, for the unknown words. The embedding layers for the encoder and for the decoder are not shared, and are learned separately. This separation is due to the words used in utterances by the customers being very different from the posts by the customer support.

For the Transformer, we use two identical layers for the encoder and for the decoder, with four heads for the self-attention. The dimensionality of the input and of the output is $d_{model}=256$ , and the inner dimensionality is $d_{inner}=512$ . The input consists of queries with keys of dimensionality $d_{k}=64$ and values of the same dimensionality $d_{v}=64$ . The input and the output embedding are learned separately with sinusoidal positional encoding.

5.4 Evaluation Measures

How to evaluate a chatbot is an open research question. As the problem is related to machine translation (MT) and text summarization (TS), which are nowadays also addressed using seq2seq models, researchers have been using MT and TS evaluation measures such as BLEU Papineni et al. (2002) and ROUGE Lin (2004), which focus primarily on word overlap and measure the similarity between the chatbot’s response and the gold answer to the user question (here, the answer by the customer support). However, it has been argued Liu et al. (2016); Lowe et al. (2017) that such word-overlap measures are not very suitable for evaluating chatbots. Thus, we adopt three additional measures, which are more semantic in nature.

The embedding average (Lowe et al., 2015) constructs a vector for a piece of text by taking the average of the word embeddings of its constituent words. Then, the vectors for the chatbot response and for the gold human answer are compared using the cosine similarity.

The greedy matching was introduced in the context of intelligent tutoring systems Rus and Lintean (2012). It matches each word in the chatbot’s output to the most similar word in the gold human response, where the similarity is measured as the cosine between the corresponding word embeddings, multiplied by a weighting term (which we set to 1), as shown in Equation (4). Since this measure is asymmetric, we also calculate it with the arguments swapped, and then we take the average as shown in Equation (5).

[TABLE]

The vector extrema (Forgues et al., 2014) was proposed for dialogue systems. Instead of averaging the word embeddings of the words in a piece of text, it takes the coordinate-wise maximum (or minimum), as shown in Equation (6). Finally, the resulting vectors for the chatbot output and for the gold human answer are compared using the cosine similarity.

[TABLE]

6 Evaluation Results

Below, we first discuss our auxiliary classification task, where the objective is to predict which question–answer pair is “good”, and then we move to the main task of answer re-ranking.

6.1 Auxiliary Task: Question–Answer Goodness Classification

Table 2 shows the results for the auxiliary task of question–answer goodness classification. The first column is the name of the model. It is followed by three columns showing the type of embedding used, the size of the hidden layer, and the number of heads (see Section 3.2). The last column reports the accuracy. Since our dataset is balanced (we generate about 50% positive, and about 50% negative examples), accuracy is a suitable evaluation measure for this task. The top row of the table shows the performance for a majority class baseline. The following lines show the results for our full QANet-based model when using different kinds of embeddings. We can see that contextualized sentence-level embeddings are preferable to using simple word embeddings as in GloVe or token-level ELMo embeddings. Moreover, while token-level ELMo outperforms GloVe when the size of the network is small, there is no much difference when the number of parameters grows ( $d_{model}=128$ , $\#Heads=8$ ).

6.2 Answer Selection/Generation: Individual Models

Table 3 reports the performance of the individual models: information retrieval (IR), Sequence-to-sequence (seq2seq), and the Transformer (see Section 5.3 for more details about these models). In our earlier work (Hardalov et al., 2018), we performed these experiments using exactly the same experimental setup. The table is organized as follows: The first column contains the name of the model used to obtain the best answer. The second and the third columns report the word overlap measures: (i) BLEU@2, which uses uni-gram and bi-gram matches between the hypothesis and the reference sentence, and (ii) ROUGE-L Lin and Och (2004), which uses Longest Common Subsequence (LCS). The last three columns are for the semantic similarity measures: (i) Embedding Average (Emb Avg) with cosine similarity, (ii) Greedy Matching (Greedy Match), and (iii) Vector Extrema (Vec Extr) with cosine similarity. In the three latter measures, we used the standard pre-trained word2vec embeddings because they are not learned during training, which helps avoid bias, as has been suggested in Liu et al. (2016); Lowe et al. (2017).

We can see in Table 3 that the seq2seq model outperforms IR by a margin on all five evaluation measures, which is consistent with previous results in the literature. What is surprising, however, is the relatively poor performance for the Transformer, which trails behind the seq2seq model on all evaluation measures. We hypothesize that this is due to the Transformer having to learn more parameters as it operates with higher-dimensional word embeddings. Overall, the Transformer is arguably slightly better than the IR model, outperforming it on three of the five evaluation measures.

The last row of Table 3 is not an individual model; it is our re-ranker applied to the top answers returned by the IR model. In particular, we use QANet with Sentence level ELMo ( $d_{model}=128$ , $\#Heads=8$ ). We took the top-5 answer candidates (the value of 5 was found using cross-validation on the training dataset) from the IR model, and we selected the best answer based on our re-ranker’s scores. We can see that re-ranking yields improvements for all evaluation measures: $+1.18$ on BLEU@2, $+0.93$ on ROUGE_L, $+1.12$ on Embedding Average, $+0.67$ on Greedy Matching, and +1.64 in Vector Extrema. These results show that we can get sizable performance gains when re-ranking the top- $K$ predictions of a single model; below we will combine multiple models.

6.3 Main Task: Multi-Source Answer Re-Ranking

Next, we combine the top- $K$ answers from different models: IR and seq2seq. We did not include the Transformer in the mix as its output is generative and similar to that of the seq2seq model; moreover, as we have seen in Table 3 above, it performs worse than seq2seq on our dataset. We set $K=2$ for the baseline, Random Top Answer, which selects a random answer from the union of the top $K$ answers by the models involved in the re-ranking. For the remaining re-ranking experiments, we use $K=5$ . We found these values using cross-validation on the training dataset, trying 1–5.

The results are shown in Table 4, where different representations are separated by a horizontal line. The first row of each group contains the name of the model. Then, on the even rows (second, forth, etc.), we show the results from a greedy answer selection strategy, while on the odd rows are the results from an exploration strategy (softmax sampling). Since softmax sampling and random selection are stochastic in nature, we include a $95\%$ confidence interval for them.

We can see in Table 4 that QANet with sentence-level ELMo ( $d_{model}=128$ , $\#Heads=8$ ) performs best in terms of BLEU@2, ROUGE_L, and Greedy Matching. Note also the correlation between higher results on the auxiliary task (see Table 2) and improvement in terms of word-overlap measures, where we find the largest difference between individual and re-ranked models ( $+1.5$ points absolute over the baseline, and $+0.95$ over seq2seq in terms of BLEU@2). In terms of semantic similarity, we note the highest increase for Embedding Average ( $+1.3$ over the baseline, and $+1.4$ over seq2seq), and a smaller one for Greedy Matching ( $+1.0$ over the baseline, and $+0.4$ over seq2seq), and Vector Extrema ( $+2.6$ over the baseline, and $+0.6$ over seq2seq).

Overall, the re-ranked models are superior as evaluated on word-matching measures, which is supported by the improvement of BLEU@2 and Embedding Average. The smaller improvement for Greedy Matching and Vector Extrema can be explained by the training procedure for the re-ranking model, which is based on word comparison. However, these two measures focus on keyword similarity between the target and the proposed answers, and generative models are better at this. This is supported by comparing the combined model to IR-BM25, where we see sizable improvements of $+1.5$ and $+2.0$ in terms of Greedy Matching and Vector Extrema, respectively.

We can further see in Table 4 that using a stochastic approach to select the best answer yields additional improvements. This strategy accounts for the predicted goodness score for each candidate, thus, enriching the model in two ways. First, implicit voting is used, as duplicate answer candidates are not removed, resulting in higher selection probability of popular answers from different input modules. Second, albeit two answers may have a very different structure, they still can be similar in meaning, leading to very similar scores and promoting only the first one. This behavior can be mitigated by choosing the winner proportionally to its ranking, thus, also introducing diversity in the chatbot’s language. This hypothesis is supported by the results in Table 4: compare each model to the corresponding one with softmax selection.

7 Conclusions and Future Work

We have presented a novel framework for re-ranking answer candidates for conversational agents. In particular, we adopted techniques from the domain of machine reading comprehension Chen et al. (2017); Seo et al. (2017); Yu et al. (2018) to evaluate the quality of a question–answer pair. Our framework consists of two tasks: (i) an auxiliary one, aiming to fit a goodness classifier using QANet and negative sampling, and (ii) a main task that re-ranks answer candidates using the learned model. We further experimented with different model sizes and two types of embedding models: GloVe Pennington et al. (2014) and ELMo Peters et al. (2018). Our experiments showed improvements in answer quality in terms of word-overlap and semantics when re-ranking using the auxiliary model. Last but not least, we argued that choosing the top-ranked answer is not always the best option. Thus, we introduced probabilistic sampling that aims to diversify the agent’s language and to up-vote the popular answers, while taking their ranking scores into consideration.

In future work, we plan to experiment with different exploration strategies such as Boltzmann exploration (softmax is a degraded version of Boltzmann exploration with $\tau=1$ ). $\epsilon$ -Confident, Upper confidence bound, and other bandit methods Sutton and Barto (1998) to widen the possible context for each answer over time. We see an interesting research direction in applying deep reinforcement learning (i) to improve the answer selection models when applied to unseen questions, and (ii) to account for user feedback and customer support task success.

\authorcontributions

Conceptualization, M.H., P.N. and I.K.; Investigation, M.H; Writing–original draft, M.H.; Writing–review & editing, P.N. and I.K.

\funding

This research is partially supported by Project UNITe BG05M2OP001-1.001-0004 funded by the OP “Science and Education for Smart Growth”, co-funded by the EU through the ESI Funds.

\conflictsofinterest

The authors declare no conflict of interest.

\reftitle

References

Bibliography48

The reference list from the paper itself. Each links out to its DOI / PubMed record.

1Yu et al. (2018) Yu, A.W.; Dohan, D.; Luong, M.T.; Zhao, R.; Chen, K.; Norouzi, M.; Le, Q.V. QA Net: Combining Local Convolution with Global Self-Attention for Reading Comprehension. In Proceedings of the 2018 International Conference on Learning Representations; Vancouver, BC, Canada, 30 April–3 May 2018.
2Seo et al. (2017) Seo, M.; Kembhavi, A.; Farhadi, A.; Hajishirzi, H. Bi-directional attention flow for machine comprehension. In Proceedings of the 2017 International Conference on Learning Representations; Toulon, France, 24–26 April 2017.
3Chen et al. (2017) Chen, D.; Fisch, A.; Weston, J.; Bordes, A. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada, 30 July–4 August 2017; pp. 1870–1879.
4Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; Zhu, W.J. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA, 7–12 July 2002; pp. 311–318.
5Lin (2004) Lin, C.Y. ROUGE: A Package for Automatic Evaluation of Summaries. In Proceedings of the ACL Workshop on Text Summarization Branches Out, Barcelona, Spain, 25–26 July, 2004; pp. 74–81.
6Lowe et al. (2015) Lowe, R.; Pow, N.; Serban, I.; Pineau, J. The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Prague, Czech Republic, 2–4 September 2015; pp. 285–294.
7Lison and Tiedemann (2016) Lison, P.; Tiedemann, J. Open Subtitles 2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation, Portorož, Slovenia, 23–28 May 2016.
8Reddy et al. (2018) Reddy, S.; Chen, D.; Manning, C.D. Co QA: A conversational question answering challenge. ar Xiv 2018 , ar Xiv:1808.07042.