Skill Check: Some Considerations on the Evaluation of Gamemastering Models for Role-playing Games

TL;DR

This paper discusses the challenges in modeling Game Masters for role-playing games using AI, proposes evaluation categories, and tests existing AI models like ChatGPT and Bard as GMs.

Contribution

It introduces a framework for evaluating AI-based Game Master models and provides empirical testing of popular language models in this role.

Findings

ChatGPT performs comparably to specialized GMs

Evaluation categories reveal strengths and weaknesses of models

Proposes future directions for AI-driven role-playing game GMs

Abstract

In role-playing games a Game Master (GM) is the player in charge of the game, who must design the challenges the players face and narrate the outcomes of their actions. In this work we discuss some challenges to model GMs from an Interactive Storytelling and Natural Language Processing perspective. Following those challenges we propose three test categories to evaluate such dialogue systems, and we use them to test ChatGPT, Bard and OpenAssistant as out-of-the-box GMs.