ProKG-Dial: Progressive Multi-Turn Dialogue Construction with Domain Knowledge Graphs

TL;DR

ProKG Dial is a progressive framework that constructs high-quality, domain-specific multi-turn dialogues using knowledge graphs, enhancing dialogue relevance, coherence, and domain coverage for specialized conversational systems.

Contribution

It introduces a novel, knowledge graph-based, progressive dialogue construction method that improves domain coverage and dialogue quality compared to existing resource-intensive approaches.

Findings

Generated dialogues show higher diversity and coherence.

Fine-tuned LLMs on ProKG Dial data outperform baselines.

Automatic and human evaluations confirm improved dialogue quality.

Abstract

Current large language models (LLMs) excel at general NLP tasks but often lack domain specific precision in professional settings. Building a high quality domain specific multi turn dialogue dataset is essential for developing specialized conversational systems. However, existing methods such as manual annotation, simulated human LLM interactions, and role based LLM dialogues are resource intensive or suffer from limitations in dialogue quality and domain coverage. To address these challenges, we introduce ProKG Dial, a progressive framework for constructing knowledge intensive multi turn dialogue datasets using domain specific knowledge graphs (KGs). ProKG Dial leverages the structured nature of KGs to encode complex domain knowledge and relationships, providing a solid foundation for generating meaningful and coherent dialogues. Specifically, ProKG Dial begins by applying community detection to partition the KG into semantically cohesive subgraphs. For each subgraph, the framework incrementally generates a series of questions and answers centered around a target entity, ensuring relevance and coverage. A rigorous filtering step is employed to maintain high dialogue quality. We validate ProKG Dial on a medical knowledge graph by evaluating the generated dialogues in terms of diversity, semantic coherence, and entity coverage. Furthermore, we fine tune a base LLM on the resulting dataset and benchmark it against several baselines. Both automatic metrics and human evaluations demonstrate that ProKG Dial substantially improves dialogue quality and domain specific performance, highlighting its effectiveness and practical utility.