ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Models

TL;DR

ConvKGYarn is a scalable, configurable method for generating up-to-date conversational Knowledge Graph QA datasets, enabling better training and evaluation of Large Language Models in dynamic, diverse interaction scenarios.

Contribution

It introduces a novel scalable approach for creating high-quality, configurable conversational KGQA datasets that adapt to evolving user information needs.

Findings

High-quality datasets comparable to existing ones

Effective testing of LLMs on diverse conversational configurations

Enhanced evaluation of LLMs' parametric knowledge

Abstract

The rapid advancement of Large Language Models (LLMs) and conversational assistants necessitates dynamic, scalable, and configurable conversational datasets for training and evaluation. These datasets must accommodate diverse user interaction modes, including text and voice, each presenting unique modeling challenges. Knowledge Graphs (KGs), with their structured and evolving nature, offer an ideal foundation for current and precise knowledge. Although human-curated KG-based conversational datasets exist, they struggle to keep pace with the rapidly changing user information needs. We present ConvKGYarn, a scalable method for generating up-to-date and configurable conversational KGQA datasets. Qualitative psychometric analyses confirm our method can generate high-quality datasets rivaling a popular conversational KGQA dataset while offering it at scale and covering a wide range of human-interaction configurations. We showcase its utility by testing LLMs on diverse conversations - exploring model behavior on conversational KGQA sets with different configurations grounded in the same KG fact set. Our results highlight the ability of ConvKGYarn to improve KGQA foundations and evaluate parametric knowledge of LLMs, thus offering a robust solution to the constantly evolving landscape of conversational assistants.