Steps towards prompt-based creation of virtual worlds

TL;DR

This paper explores prompt-based methods for creating and editing virtual worlds using large language models, demonstrating their potential in gameplay and level design, and discussing evaluation challenges.

Contribution

It introduces a novel approach to using prompt-based generative models for VR content creation and gameplay, including a prototype game called Codex VR Pong.

Findings

Prompt-based methods accelerate VR level editing.

Generative models enable non-trivial interactions in VR.

Discussion on the lack of evaluation metrics for AI-generated VR experiences.

Abstract

Large language models trained for code generation can be applied to speaking virtual worlds into existence (creating virtual worlds). In this work we show that prompt-based methods can both accelerate in-VR level editing, as well as can become part of gameplay rather than just part of game development. As an example, we present Codex VR Pong which shows non-deterministic game mechanics using generative processes to not only create static content but also non-trivial interactions between 3D objects. This demonstration naturally leads to an integral discussion on how one would evaluate and benchmark experiences created by generative models - as there are no qualitative or quantitative metrics that apply in these scenarios. We conclude by discussing impending challenges of AI-assisted co-creation in VR.