Bridging Information Gaps with Comprehensive Answers: Improving the Diversity and Informativeness of Follow-Up Questions

TL;DR

This paper introduces a knowledge distillation pipeline that enhances small conversational models with diverse, informative follow-up questions by leveraging a teacher LLM to identify information gaps and generate comprehensive training data.

Contribution

The authors develop an information-gap-driven distillation method that significantly improves the diversity and informativeness of follow-up questions in small models.

Findings

Fine-tuned models outperform original datasets in diversity and informativeness.

Augmented dataset increases the quality of follow-up questions by tenfold.

The pipeline effectively transfers knowledge from large to small models.

Abstract

Generating diverse follow-up questions that uncover missing information remains challenging for conversational agents, particularly when they run on small, locally hosted models. To address this, we develop an information-gap-driven knowledge distillation pipeline in which a teacher LLM generates a comprehensive answer, contrasts it with the initial answer to identify information gaps, and formulates gap-bridging follow-up questions. Using this pipeline, we augment the existing FollowupQG dataset tenfold. We then fine-tune smaller student models on the augmented dataset to distill the teacher's knowledge. Experiments with selected teacher-student model pairs show that fine-tuned students achieve significantly higher informativeness and diversity than variations trained on the original dataset. These findings indicate that our pipeline, which mirrors the human cognitive process of information seeking, provides an efficient distillation channel from state-of-the-art LLMs to smaller models, enabling resource-constrained conversational systems to generate more diverse and informative follow-up questions.