Conversational Help for Task Completion and Feature Discovery in Personal Assistants

TL;DR

This paper presents an interactive system for personal assistants that detects help queries and provides relevant responses, improving user understanding of available skills and commands through a neural network-based classifier and semantic matching.

Contribution

It introduces a novel C-BiLSTM classifier combined with semantic ANN for effective help query detection and response retrieval in IPAs, outperforming existing models.

Findings

System outperforms traditional models in help query detection.

Accurately retrieves relevant responses for user help queries.

Effective in real-world commercial IPA scenarios.

Abstract

Intelligent Personal Assistants (IPAs) have become widely popular in recent times. Most of the commercial IPAs today support a wide range of skills including Alarms, Reminders, Weather Updates, Music, News, Factual Questioning-Answering, etc. The list grows every day, making it difficult to remember the command structures needed to execute various tasks. An IPA must have the ability to communicate information about supported skills and direct users towards the right commands needed to execute them. Users interact with personal assistants in natural language. A query is defined to be a Help Query if it seeks information about a personal assistant's capabilities, or asks for instructions to execute a task. In this paper, we propose an interactive system which identifies help queries and retrieves appropriate responses. Our system comprises of a C-BiLSTM based classifier, which is a fusion of Convolutional Neural Networks (CNN) and Bidirectional LSTM (BiLSTM) architectures, to detect help queries and a semantic Approximate Nearest Neighbours (ANN) module to map the query to an appropriate predefined response. Evaluation of our system on real-world queries from a commercial IPA and a detailed comparison with popular traditional machine learning and deep learning based models reveal that our system outperforms other approaches and returns relevant responses for help queries.

Figures

Tables

Table 1. Table 1: Possible Help intents and sample queries

Intent	Sample	Query
	Query	Type
Overall capabilities	What can	Generic
of the IPA	you do?
Information about	Do you have	Skill
a particular skill	flight tracking?
Direct help	Help me to	Skill
for a task	setup an alarm.
Questions on task	Tell me how to	Skill
instructions	connect via bluetooth?
Interest in a task	I would like	Skill
	to play music.

Table 2. Table 2: Optimal Hyper-Parameters for various deep learning models, as tuned on the Validation Set

Parameter	LSTM	BiLSTM	C-BiLSTM
Embd. Size	300	300	300
MaxLen	15	15	15
Batch Size	1024	1024	1024
Filter Size	NA	NA	128
LSTM cells	32	32	32
Optimizer	Adagrad	Adagrad	Adagrad
Learning rate	0.001	0.001	0.001

Table 3. Table 3: Classification performance of various models on the Test set. C-BiLSTM is found to outperform other models in detecting Help queries.

Model	Precision	Recall	F1 Score
XGBoost	0.844	0.809	0.824
SVM	0.916	0.789	0.848
CNN	0.914	0.807	0.857
LSTM	0.906	0.810	0.855
BiLSTM	0.92	0.819	0.867
C-BiLSTM	0.952	0.826	0.885

Table 4. Table 4: Qualitative analysis of the POS-based response fetching module. This approach works well for simple help queries but fails for generic and unseen help queries.

	Action/
Query	Skill	Comments
How to connect	Connect/	Mapped to correct
via bluetooth?	Bluetooth	response.
Can you hook up	Hook up/	Hook up was not listed
via bluetooth?	Bluetooth	as an action, resulting
		in recall gap.
Tell me the steps to	Sync/	Sync is not added
sync my smart tv.	TV	to list of actions.
What can	Do/	This approach doesn’t
you do?	-	work for generic help
		queries, as there is
		no skill mentioned.

Table 5. Table 5: Precision vs Recall trade-off for various similarity thresholds in semantic query matching scores.

Similarity
Threshold		Precision	Recall	F1 Score
0.85		0.924	0.633	0.751
0.83		0.904	0.679	0.776
0.82		0.896	0.699	0.785
0.80		0.878	0.728	0.796
0.75		0.843	0.762	0.801
0.70		0.752	0.791	0.771
0.65		0.712	0.829	0.766
0.60		0.667	0.850	0.745

Table 6. Table 6: Comparison of POS-based and DSSM based response matching approaches.

Module	Precision	Recall	F1 Score
POS-based action-skill pairs	0.89	0.61	0.72
DSSM-based ANNs	0.81	0.74	0.77

Full text

Users interact with personal assistants in natural language. A query is defined to be a Help Query if it seeks information about a personal assistant’s capabilities, or asks for instructions to execute a task. In this paper, we propose an interactive system which identifies help queries and retrieves appropriate responses. Our system comprises of a C-BiLSTM based classifier, which is a fusion of Convolutional Neural Networks (CNN) and Bi-directional LSTM (BiLSTM) architectures, to detect help queries and a semantic Approximate Nearest Neighbours (ANN) module to map the query to an appropriate predefined response. Evaluation of our system on real-world queries from a commercial IPA and a detailed comparison with popular traditional machine learning and deep learning based models reveal that our system outperforms other approaches and returns relevant responses for help queries.

1 Introduction

In the era of Artificial Intelligence (AI), IPAs have gained a lot of attention and found their way into millions of homes. Google Now from Google, Siri from Apple, Alexa from Amazon and Cortana from Microsoft are the four major commercial IPAs in market today. All of them support a wide range of tasks including setting up alarms, getting weather updates, tracking flights, playing music, etc, and are adding many other skills and scenarios every day.

However, skill expansion brings with it multiple unique challenges. First of all, feature discovery becomes a critical factor. As the system evolves continuously, it is difficult for the users to stay updated with the latest and upcoming skills. The most common and widely used way of solving this problem is to have a help page which states all the existing and new skills. Other approaches include building a FAQ bot which can respond to factual user queries. However, these are very naive and non-intuitive approaches. With most of the IPAs having a voice-enabled interface, a more interactive ecosystem is needed where users interact with the personal assistant in natural language to discover its new features. Secondly, every skill has its own functionality and a set commands that it supports. For example, a “Weather” skill which provides users with the weather updates and forecast from any part of the world is reasonably different from an “Alarm” skill which allows users to create, delete or snooze alarms. These skills serve very different purposes and have their own independent set of commands. Therefore, IPAs need to have a system to guide users towards the right command structures needed to execute any task and educate them about the overall utility of a skill. Lastly, for a smooth, fluent and satisfactory user experience, a personal assistant should assist the users in completing the intended tasks and guide them to use the right set of commands in a seamless manner.

A query is defined to be a Help if it’s intent is one of the following – a) asks for the IPA’s overall capabilities b) seeks information about a particular skill c) directly seeks help for executing a specific task d) questions about instructions required to complete a task e) showing interest in any task. Based on the intent, help queries can broadly be divided into two categories – skill help and generic help. Queries which seek information about the overall capabilities of the system, intent a), fall under the generic help category. On the other hand, queries that seek information about a particular skill of the system, intents b)-e), are defined as skill/task help queries. Table 1 tabulates some sample queries with each of the above defined intents.

In this paper, we propose a system which identifies help queries and provides appropriate responses. The system is made up of two modules. The first module uses C-BiLSTM deep learning architecture, which is a fusion of Convolutional Neural Networks (CNN) and Bi-directional LSTMs (BiLSTM), to detect help queries. Given an input query, C-BiLSTM first passes it through a convolution layer to learn various important local features, which in turn are passed through a BiLSTM layer which encodes various sequential information in the query by making a forward as well as backward pass over the query features. The deep features, thus obtained, are fed into a Fully Connected (FC) layer to predict whether the query belongs to help class or not. We have also compared this model with various hand-crafted features based traditional machine learning models, and deep learning based individual CNN, LSTM and BiLSTM models. Results on a real-world help query dataset reveals that C-BiLSTM outperforms other models.

To map the identified help query to one of the predefined responses, we leverage the labelled training data to find the queries which are semantically similar to the input query. To achieve this in real-time, we use KDTrees-based Approximated Nearest Neighbours (ANN) (Panigrahy (2008)) approach to find top-k similar help queries using their DSSM (Huang et al. (2013)) feature vectors. We return the response of the closest matched query if the corresponding response is in majority and the similarity score is above a certain threshold.

In summary, the following are our main contributions in this paper:

• •

We introduce the research problem of conversational help systems for task completion and feature discovery in personal assistants.

• •

We propose an interactive conversational help system which detects help queries and provides appropriate responses.

• •

We evaluated and compared various latest deep learning techniques including CNN, LSTM, BiLSTM and C-BiLSTM for classifying help queries.

• •

We used KDTree-based ANNs on DSSM semantic query features to find similar queries in real-time and use them to map the input query to an appropriate response.

The rest of the paper is organized as follows. Section 2 presents previous works done in the field of personal assistants and conversational task completion systems. Section 3 describes our approach for building the system. Section 4 describes the experimental setup, details of the dataset used and various quantitative and qualitative results. Section 5 concludes the paper.

2 Related Work

Ever since the evolution of personal assistants, various studies pertaining to user understanding, context aware conversations and recommendations, etc., have been carried out in literature. Guha et al. (2015) proposed a system that uses web search history to identify coherent contexts that correspond to tasks, interests, and habits of the signed-in Google Now users. They look at several months of history in order to identify not just short-term tasks, but also long-term interests and habits of the users. Elwany, Shakeri (2014) analyzed millions of user queries from Cortana to build a machine learning system capable of classifying user queries into two classes – a class of queries that are addressable by Cortana with high user satisfaction, and a class of queries that are not. Kumar, Joshi (2017) proposed a retrieval-based sequence to sequence learning system that generates complete (intended) question for an incomplete follow-up question, given the conversation context, in Siri. Lopatovska et al. (2018) examined user interactions with Alexa, and focused on the types of tasks requested of Alexa, the variables that affect user behaviors with Alexa, and Alexa’s alternatives.

In some other works, interactive systems have been deployed to provide conversational assistance in completing tasks. Thompson et al. (2004) proposed a system that treats item selection as an interactive, conversational process, with the program inquiring about item attributes and the user responding. Levinson, Shipley (1980) designed a conversational-mode, speech-understanding system which enables its user to make airline reservations and obtain timetable information through a spoken dialog. Janarthanam et al. (2013) demonstrated a conversational interface that assists pedestrian users in navigating within urban environments and acquiring tourist information by combining spoken dialogue system, question-answering, and geographic information system technologies. Some progress have been done in building question answering system from FAQ (Frequently Asked Questions) files. Burke et al. (1997) used a purely lexical metric of similarity between query and document and a semantic knowledge base (WordNet) to match question and answer from FAQ files. Calculation of question similarity is a key problem in FAQ answering which was solved using question vector similarity based on WordNet (Song et al. (2007)) and using similarity between answers in the archive to estimate probabilities for a translation-based retrieval model (Jeon et al. (2005)).

However, to the best of our knowledge, there is no prior work related to conversational help systems for task completion and feature discovery in personal assistants. In this paper, we propose a two-level system where we first classify help queries, and then use semantic similarity between queries to map the input query to an appropriate response.

Over the last decade, deep learning based models have achieved great success in many of the NLP tasks, including sentence classification. Kim (2014) showed that a simple CNN with little hyper-parameter tuning and static vectors achieves excellent results, and improved upon the state-of-the-art on 4 bench-marking tasks including sentiment analysis and question classification. On the other hand, RNNs with Long Short-Term Memory units Mikolov (2012); Chung et al. (2014); Tai et al. (2015) are effective networks to process sequential data, which analyze a text word by word and stores the semantics of all the previous text in a fixed-sized hidden state. In this way, LSTM can better capture the contextual information and semantics of long texts. Moreover, bidirectional RNNs (Schuster, Paliwal (1997)) processes the sequence both forward and backward, naturally, a better semantic representation can usually be obtained than unidirectional RNNs. Zhou et al. (2015) combined CNN and LSTM architectures for sentence representation and text classification. Recently, Yenala et al. (2017) combined the strengths of CNNs and BiLSTMs and proposed C-BiLSTM to classify inappropriate queries on real-world web search dataset. They evaluated other traditional as well as deep learning based classifiers and showed that C-BiLSTM performed better than individual CNN, LSTM and BiSLTM models.

Huang et al. (2013) proposed DSSM, a deep neural network modeling technique for representing text strings (sentences, queries, predicates, entity mentions, etc.) in a continuous semantic space. DSSM can be used to develop latent semantic models that project multiple entities into a common low-dimensional semantic space for a variety of machine learning tasks such as ranking and classification. For example, for finding similar queries, the relevance between two queries can be readily computed as the cosine similarity between them in that space. The DSSM similarity score is found to be better than many traditional features including TF-IDF, BM25, Word Translation Model (WTM), LSA, PLSA, and LDA, etc. (Ye et al. (2015); Bogdanova, Yazdani (2017)), and has been widely used for various semantic similarity based tasks (Ye et al. (2016); Palangi et al. (2014)). For the task of fetching appropriate response, we used DSSM based semantic similarity between queries.

3 Conversational Help System

The architecture of the conversational help system is depicted in Figure 1. The system is divided into two modules – Help Query Classification and Response Mapping.

A fusion model which combines the strengths of convolutional neural networks and bidirectional LSTMs is used to detect help queries. The model takes the word embedding matrix of the query as input and passes it through three sequential layers – a) Convolutional (CONV) Layer, b) Bi-directional LSTM (BLSTM) Layer, and c) Fully Connected (FC) Layer. The CONV layer first project the input query embedding matrix into a lower-dimensional feature representation. The BLSTM layer encodes sequential information by making forward and backward passes over the query features. The FC layer finally models the query features and outputs whether it is a help query or not.

Once the input query is identified as a help query, a distributed query similarity model is used to fetch the appropriate response. We use deep semantic features of the input query to find top-k similar help queries from the training data, which have been cached offline. We use majority voting to pick the most appropriate response suitable for the input query.

3.1 Help Query Classification

In this section, we will explain the working of help classifier in detail.

3.1.1 Query Normalization

Real-world user queries come with a lot of noise in the form of spelling mistakes, incorrect sentences, use of slang words and phrases, short-forms, etc. Therefore, we have carried out several domain-specific as well as standard normalization steps. We convert the input query to lower case and remove stop words and punctuation marks, followed by lemmatization. We translate more than 50 most frequently used slang words and phrases, such as wanna, gimme, etc., to their English counterparts. Since Alarms, Reminders and Timers are among the supported skills, we observed that a good portion of the queries contained time stamps. Some of the examples of such queries are – “how to set an alarm for 5:00pm”, “help me to cancel the 10:00pm reminder for tomorrow”, etc. Therefore, we used regular expressions to convert all the time stamps with the token “time_stamp”. Similarly, Music and Radio are other top skills. Hence, we mined top-300 music genres and replaced them with the token “music_genre”.

3.1.2 Query Embedding

DSSM (Huang et al. (2013)) has been trained on real-user queries which makes it our first choice for word embedding. Furthermore, it uses character-trigrams which enables it to generate embedding for misspelled and unseen words. Therefore, DSSM word embedding is used to project every word in the input query into a 300-dimensional feature space, forming the query matrix. As the length of the input queries could vary, we pad them with “unk” tokens, representing unknown words, at the beginning to fix the input query length to maxlen (15 in our case).

3.1.3 The Convolutional Layer

The representation of the input query is given by a matrix $Q\in R^{d*l}$, where $d$ is the dimensionality of word embedding and $l$ is the length of the input query (maxlen). A 1-D convolution layer is applied on the word embedding matrix $Q$ by convolving a filter $H\in R^{d*m}$ of length $m$, followed by a max-pooling layer to extract the important local features. An element-wise non-linear transformation using Rectified Linear Unit (ReLU) is applied after convolution operation to avoid saturation and improve learning.

3.1.4 The BiLSTM Layer

The fixed length feature vector obtained from the convolution layer is fed into the BLSTM layer. Bi-directional LSTMs make forward and backward passes over the input and capture relevant sequential patterns. The outputs of the forward and backward LSTM cells inside the BLSTM are concatenated to form a combined feature output encoding rich semantic features of the query. These features are passed onto a FC layer which models the various interactions between the features. The final softmax node in the FC layer outputs the probability of the query belonging to the help class.

3.2 Response Fetching

Cortana today supports a wide range of tasks, such as playing music, creating alarms, tracking packages, etc. For each of such tasks, for which help has to be provided, we used human experts to get an appropriate response string which can guide the users towards completing the task.

After classifying an input query as a help query, we need to map it to one of the predefined responses. Since a help query can be framed in multiple ways, understanding the semantic meaning of the query becomes an important part. For example, to ask help for deleting an alarm, queries can be framed as “help me to delete my alarm”, or “how do I shut the alarm up”, or “instructions on deleting an alarm”, etc.

Therefore, we leverage the labelled training data to find top-k queries which are semantically similar to the input query. We use cosine similarity between the DSSM features (DSSM model used is pretained with query-document pairs of a commercial search engine) of queries as the measure of semantic similarity between them. For this, in offline, we map all the help queries in our training data, a total of 50K queries, to their DSSM feature vectors. We store these feature vectors, along with their appropriate responses, in a cache so that they can be readily used in real-time. For the input query, we use its DSSM features to find its top-k (k=1 in our case) ANNs, using KDTrees (Panigrahy (2008)), from cached feature vectors. The ANN algorithm finds approximately similar queries with a very low latency. If the similarity between the closest query and the input query is above a certain threshold, we return the response corresponding to the matched query.

4 Experiments and Results

In this section, we describe the dataset used, the quantitative results of various experiments we have carried out with feature and model selection for the help query classifier. Furthermore, we explain the challenges, experiments, and quantitative and qualitative results of the response mapping approaches.

4.1 Dataset

We have mined a dataset comprising of 200K unique queries from anonymized and privacy preserved Cortana logs of internal users. The data has been manually labeled by human judges into two categories – help or not help. In case of help, they also tagged the queries with one of the predefined responses. To avoid human bias and judgment errors, each query was judged by three judges. The judges were given clear guidelines, along with sample query-label pairs, on how to classify the queries. A query was finally labeled help if majority of the judges labeled it as help. The dataset comprises of 24 different supported skills and varying tasks. The distribution of the skills in the dataset is depicted in Figure 2. We have randomly divided the data into train, validation and test sets. The training set comprises of $80\%$ of queries, while the validation and test sets comprises of $5\%$ and $15\%$ of queries respectively.

4.2 Help Query Classification

For detecting help queries, we experimented with both hand-crafted features based as well as deep learning based classifiers. We used word-based unigram and bigram features to evaluate two widely used traditional machine learning algorithms – Support Vector Machine (SVM) and Gradient Boosted Decision Trees (XGBoost). We found that XGBoost yielded slightly better recall at the cost of lower precision in classifying help queries. Among the deep learning based classifiers, we evaluated the most popular classifiers including CNN based classifier, LSTM and BiLSTM based classifiers and compared them with C-BiLSTM model for detecting help queries. We used DSSM word embedding query matrix as the input features for all the deep learning based classifiers. The optimal hyper-parameters for various classifiers, as tuned on the Validation set, is tabulated in Table 2.

Table 3 tabulates the precision, recall and F1-scores of the various models on the Test set. As it can be seen, all the deep learning based classifiers are able to outperform the traditional ngram-based classification algorithms. Also, while the LSTM and BiLSTM based classifiers produced comparable results, C-BiLSTM model yielded the best performance in detecting help queries.

4.3 Response Fetching

Help queries, by definition, seek help about tasks. A task is defined as performing an action of a skill. For example, for the query “How to create an alarm?”, alarm is the skill and create is the action. Similarly, for the query “Help me to play music.”, music is the skill and play is the action. Therefore, we experimented with mapping help responses with supported action-skill pairs. We expanded the actions list with their popular synonyms to increase the coverage. For example, queries “How to create an alarm?” and “Can you set up an alarm?” are mapped to the same action – “create”, as set up is a synonym of create.

To extract the action and skill, we used Parts-Of-Speech (POS) tags of the input query. We used the main verb of the query as the action and the noun as the skill. Thus, the query is mapped to the help response corresponding to the extracted action-skill pair. Upon evaluation on the test set, as there is no parameter tuning on the validation set, this approach yielded 0.72 F1 score, with 0.89 precision and 0.61 recall scores. A qualitative analysis of the approach and the results in tabulated in Table 4.

As it can be seen, there are three major shortcomings of this approach. First, it doesn’t not support generic help queries as they do not have associated actions and skills. Second, this approach is not scalable as maintaining a map of verbs to actions requires constant human interventions. Also, it does not automatically scale up to new skills having different sets of actions. And third, while the approach works well for simple help queries, it lacks semantic query understanding required for complex queries.

Therefore, we experimented with DSSM-based semantic similarity between input query and the existing help queries. The input query is mapped to the response corresponding to the best match. This approach helps us in overcoming all the three challenges:

• •

Training data comprises of both generic as well as task help queries, and the input query is compared with all of them.

• •

Instead of maintaining and updating an ever-lasting map of verbs to actions, we seed the existing and new skills with a few sample queries and the semantic similarity module maps the input query to an existing query. The new queries are added to the training data and corpus expands over time, further improving the performance.

• •

Any word or ngram based nearest-neighbors similarity algorithm would have faced the same challenge of unseen actions. On the other hand, DSSM based cosine similarity has been widely used for various semantic similarity based tasks (Ye et al. (2016); Palangi et al. (2014)), and helps us to map complex queries to their similar queries.

However, finding the similarity of the input query with all the other queries, which are increasing in number, adds latency and is not feasible in real-time systems like personal assistants. Therefore, we use KDTrees-based ANNs of the input query. ANNs yield reasonable results while meeting the real-time latency constraints.

The trade-off between precision and recall for various similarity thresholds on Validation set is shown in Table 5. As it can be seen, reducing the threshold yields higher recall at the cost of lower precision and vice-versa. We chose the similarity threshold to be greater than 0.75, as it gives the best F1-score of 0.801.

Table 6 compares the performance of both the approaches on Test set for response fetching. DSSM based ANN approach yields higher F1 score.

5 Conclusions and Future Work

In this paper, we introduced the research problem of detecting and understanding help queries in personal assistants. We proposed an interactive conversational help system which detects help queries and returns appropriate responses. We experimented with traditional as well as popular deep learning architectures to classify help queries and found that C-BiLSTM, a fusion of CNN and BiLSTM, yields the best results. We used DSSM-based semantic similarity to find ANNs of the input query and used them to map the input query to an appropriate predefined response. While the use case is specific, the help model is generic and can be used for queries of other domains.

As part of the future work, we would evaluate other semantic query-query and query-response similarity approaches and compare them against DSSM-based similarity. While the system works for skills with lesser number of training samples, we would like to explore models which yield better results and imporve scalability to new skills.