MoGraphGPT: Creating Interactive Scenes Using Modular LLM and Graphical Control

TL;DR

MoGraphGPT combines modular large language models with graphical controls to simplify and improve the creation of complex 2D interactive scenes without coding, offering precise, independent element editing and enhanced visual integration.

Contribution

This paper introduces a novel modular LLM framework with graphical interface for interactive scene creation, addressing errors and control limitations of traditional LLM-based code generation.

Findings

Significantly improves ease of scene creation

Enhances controllability and refinement of visual elements

Outperforms baseline in usability and precision

Abstract

Creating interactive scenes often involves complex programming tasks. Although large language models (LLMs) like ChatGPT can generate code from natural language, their output is often error-prone, particularly when scripting interactions among multiple elements. The linear conversational structure limits the editing of individual elements, and lacking graphical and precise control complicates visual integration. To address these issues, we integrate an element-level modularization technique that processes textual descriptions for individual elements through separate LLM modules, with a central module managing interactions among elements. This modular approach allows for refining each element independently. We design a graphical user interface, MoGraphGPT , which combines modular LLMs with enhanced graphical control to generate codes for 2D interactive scenes. It enables direct integration of graphical information and offers quick, precise control through automatically generated sliders. Our comparative evaluation against an AI coding tool, Cursor Composer, as the baseline system and a usability study show MoGraphGPT significantly improves easiness, controllability, and refinement in creating complex 2D interactive scenes with multiple visual elements in a coding-free manner.