DocTalk: Scalable Graph-based Dialogue Synthesis for Enhancing LLM Conversational Capabilities

TL;DR

This paper introduces DocTalk, a large-scale, graph-based dialogue dataset synthesized from Wikipedia articles, which improves LLMs' multi-turn conversational abilities by up to 40% during pre-training.

Contribution

The paper presents a novel pipeline for creating extensive multi-turn dialogue data from existing texts, enhancing LLMs' conversational skills during pre-training.

Findings

Up to 40% improvement in context memory and understanding

Synthesized dialogues do not compromise base performance

Effective pre-training data for multi-turn capabilities

Abstract

Large Language Models (LLMs) are increasingly employed in multi-turn conversational tasks, yet their pre-training data predominantly consists of continuous prose, creating a potential mismatch between required capabilities and training paradigms. We introduce a novel approach to address this discrepancy by synthesizing conversational data from existing text corpora. We present a pipeline that transforms a cluster of multiple related documents into an extended multi-turn, multi-topic information-seeking dialogue. Applying our pipeline to Wikipedia articles, we curate DocTalk, a multi-turn pre-training dialogue corpus consisting of over 730k long conversations. We hypothesize that exposure to such synthesized conversational structures during pre-training can enhance the fundamental multi-turn capabilities of LLMs, such as context memory and understanding. Empirically, we show that incorporating DocTalk during pre-training results in up to 40% gain in context memory and understanding, without compromising base performance. DocTalk is available at https://huggingface.co/datasets/AmazonScience/DocTalk.