AgenticAI-DialogGen: Topic-Guided Conversation Generation for Fine-Tuning and Evaluating Short- and Long-Term Memories of LLMs

TL;DR

This paper introduces AgenticAI-DialogGen, a modular framework that generates and evaluates topic-guided, memory-grounded conversations for LLMs without human supervision, improving memory handling and conversational quality.

Contribution

The paper presents a novel, unsupervised, agent-based framework for generating and evaluating topic-guided conversations with encoded memories, along with a new dataset for memory-grounded evaluation.

Findings

AgenticAI-DialogGen produces higher-quality conversations.

LLMs fine-tuned on TGC outperform on memory-grounded QA tasks.

The framework effectively encodes short- and long-term memories in conversations.

Abstract

Recent advancements in Large Language Models (LLMs) have improved their ability to process extended conversational contexts, yet fine-tuning and evaluating short- and long-term memories remain difficult due to the absence of datasets that encode both short- and long-term conversational history. Existing conversational datasets lack memory grounding, overlook topic continuity, or rely on costly human annotation. To address these gaps, we introduce AgenticAI-DialogGen, a modular agent-based framework that generates persona-grounded and topic-guided conversations without human supervision. The framework uses LLM agents to extract knowledge graphs, identify topics, build speaker personas, and simulate topic-guided conversations from unstructured conversations. A QA module generates memory-grounded Question Answer (QA) pairs drawn from short- and long-term conversational histories. We also generated a new dataset entitled, TopicGuidedChat (TGC), where long-term memory is encoded as speaker-specific knowledge graphs and short-term memory as newly generated topic-guided conversations. Evaluations depict that AgenticAI-DialogGen yields higher conversational quality and LLMs fine-tuned on TGC dataset achieve improved performance on memory-grounded QA tasks.