Learning to Retrieve Engaging Follow-Up Queries

TL;DR

This paper introduces a retrieval-based system and dataset for predicting engaging follow-up questions in open domain conversations, aiming to enhance user engagement and knowledge exploration.

Contribution

It presents a new dataset, FQ-Bank, with challenging negative examples, and evaluates ranking models for predicting follow-up questions in multi-turn dialogues.

Findings

Supervised models outperform unsupervised in ranking follow-up questions.

Confounder-specific techniques improve negative sample generation.

Knowledge grounding enhances ranking accuracy.

Abstract

Open domain conversational agents can answer a broad range of targeted queries. However, the sequential nature of interaction with these systems makes knowledge exploration a lengthy task which burdens the user with asking a chain of well phrased questions. In this paper, we present a retrieval based system and associated dataset for predicting the next questions that the user might have. Such a system can proactively assist users in knowledge exploration leading to a more engaging dialog. The retrieval system is trained on a dataset which contains ~14K multi-turn information-seeking conversations with a valid follow-up question and a set of invalid candidates. The invalid candidates are generated to simulate various syntactic and semantic confounders such as paraphrases, partial entity match, irrelevant entity, and ASR errors. We use confounder specific techniques to simulate these negative examples on the OR-QuAC dataset and develop a dataset called the Follow-up Query Bank (FQ-Bank). Then, we train ranking models on FQ-Bank and present results comparing supervised and unsupervised approaches. The results suggest that we can retrieve the valid follow-ups by ranking them in higher positions compared to confounders, but further knowledge grounding can improve ranking performance.