DialogueForge: LLM Simulation of Human-Chatbot Dialogue

TL;DR

DialogueForge is a framework that uses large language models to generate realistic human-chatbot dialogues, reducing manual data collection efforts and enabling better training and evaluation of conversational AI systems.

Contribution

The paper introduces DialogueForge, a novel framework for AI-simulated human-chatbot conversations, including techniques for fine-tuning smaller models to improve dialogue quality.

Findings

Large proprietary models produce more realistic dialogues.

Smaller open-source models can be improved with fine-tuning.

Maintaining coherence in long dialogues remains challenging.

Abstract

Collecting human-chatbot dialogues typically demands substantial manual effort and is time-consuming, which limits and poses challenges for research on conversational AI. In this work, we propose DialogueForge - a framework for generating AI-simulated conversations in human-chatbot style. To initialize each generated conversation, DialogueForge uses seed prompts extracted from real human-chatbot interactions. We test a variety of LLMs to simulate the human chatbot user, ranging from state-of-the-art proprietary models to small-scale open-source LLMs, and generate multi-turn dialogues tailored to specific tasks. In addition, we explore fine-tuning techniques to enhance the ability of smaller models to produce indistinguishable human-like dialogues. We evaluate the quality of the simulated conversations and compare different models using the UniEval and GTEval evaluation protocols. Our experiments show that large proprietary models (e.g., GPT-4o) generally outperform others in generating more realistic dialogues, while smaller open-source models (e.g., Llama, Mistral) offer promising performance with greater customization. We demonstrate that the performance of smaller models can be significantly improved by employing supervised fine-tuning techniques. Nevertheless, maintaining coherent and natural long-form human-like dialogues remains a common challenge across all models.