Dungeons and Dragons as a Dialog Challenge for Artificial Intelligence

TL;DR

This paper presents Dungeons and Dragons as a dialogue challenge for AI, introducing a large dataset and training models to generate game dialogue and predict game states, advancing dialogue system research.

Contribution

It creates a comprehensive D&D gameplay dataset and develops models for dialogue generation and game state prediction, framing D&D as a novel AI dialogue challenge.

Findings

Large dataset with 800,000 dialogue turns and 58 million words created.

Models can generate in-character and out-of-character dialogue effectively.

Game state tracking improves dialogue plausibility and relevance.

Abstract

AI researchers have posited Dungeons and Dragons (D&D) as a challenge problem to test systems on various language-related capabilities. In this paper, we frame D&D specifically as a dialogue system challenge, where the tasks are to both generate the next conversational turn in the game and predict the state of the game given the dialogue history. We create a gameplay dataset consisting of nearly 900 games, with a total of 7,000 players, 800,000 dialogue turns, 500,000 dice rolls, and 58 million words. We automatically annotate the data with partial state information about the game play. We train a large language model (LM) to generate the next game turn, conditioning it on different information. The LM can respond as a particular character or as the player who runs the game--i.e., the Dungeon Master (DM). It is trained to produce dialogue that is either in-character (roleplaying in the fictional world) or out-of-character (discussing rules or strategy). We perform a human evaluation to determine what factors make the generated output plausible and interesting. We further perform an automatic evaluation to determine how well the model can predict the game state given the history and examine how well tracking the game state improves its ability to produce plausible conversational output.