What should I Ask: A Knowledge-driven Approach for Follow-up Questions Generation in Conversational Surveys

TL;DR

This paper introduces a new knowledge-driven approach for generating follow-up questions in conversational surveys, enhancing their relevance and coherence through a novel dataset and evaluation metrics.

Contribution

It presents a new task, dataset, evaluation metrics, and a two-staged model for knowledge-driven follow-up question generation in surveys.

Findings

Our model outperforms GPT-based baselines in question informativeness and coherence.

The dataset enables systematic evaluation of follow-up question quality.

Proposed metrics correlate well with human judgments.

Abstract

Generating follow-up questions on the fly could significantly improve conversational survey quality and user experiences by enabling a more dynamic and personalized survey structure. In this paper, we proposed a novel task for knowledge-driven follow-up question generation in conversational surveys. We constructed a new human-annotated dataset of human-written follow-up questions with dialogue history and labeled knowledge in the context of conversational surveys. Along with the dataset, we designed and validated a set of reference-free Gricean-inspired evaluation metrics to systematically evaluate the quality of generated follow-up questions. We then propose a two-staged knowledge-driven model for the task, which generates informative and coherent follow-up questions by using knowledge to steer the generation process. The experiments demonstrate that compared to GPT-based baseline models, our two-staged model generates more informative, coherent, and clear follow-up questions.