Dialogue Benchmark Generation from Knowledge Graphs with Cost-Effective Retrieval-Augmented LLMs

TL;DR

This paper presents Chatty-Gen, a cost-effective, multi-stage retrieval-augmented system that automatically generates high-quality, domain-specific dialogue benchmarks from knowledge graphs, improving over existing methods in efficiency and consistency.

Contribution

Introduces Chatty-Gen, a novel multi-stage retrieval-augmented framework for automated dialogue benchmark generation from knowledge graphs, reducing reliance on costly LLMs and ensuring quality control.

Findings

Outperforms state-of-the-art systems in benchmark quality

Works effectively across diverse large knowledge graphs

Maintains performance across various LLMs like GPT-4o, Gemini 1.5, Llama 3, and Mistral

Abstract

Dialogue benchmarks are crucial in training and evaluating chatbots engaging in domain-specific conversations. Knowledge graphs (KGs) represent semantically rich and well-organized data spanning various domains, such as DBLP, DBpedia, and YAGO. Traditionally, dialogue benchmarks have been manually created from documents, neglecting the potential of KGs in automating this process. Some question-answering benchmarks are automatically generated using extensive preprocessing from KGs, but they do not support dialogue generation. This paper introduces Chatty-Gen, a novel multi-stage retrieval-augmented generation platform for automatically generating high-quality dialogue benchmarks tailored to a specific domain using a KG. Chatty-Gen decomposes the generation process into manageable stages and uses assertion rules for automatic validation between stages. Our approach enables control over intermediate results to prevent time-consuming restarts due to hallucinations. It also reduces reliance on costly and more powerful commercial LLMs. Chatty-Gen eliminates upfront processing of the entire KG using efficient query-based retrieval to find representative subgraphs based on the dialogue context. Our experiments with several real and large KGs demonstrate that Chatty-Gen significantly outperforms state-of-the-art systems and ensures consistent model and system performance across multiple LLMs of diverse capabilities, such as GPT-4o, Gemini 1.5, Llama 3, and Mistral.