Closing the Loop: Unified 3D Scene Generation and Immersive Interaction via LLM-RL Coupling

TL;DR

This paper introduces a unified framework that integrates language-driven 3D scene generation with immersive user interaction, enhancing responsiveness and realism in multimedia systems.

Contribution

It presents a novel closed-loop system coupling large language models with reinforcement learning for adaptive 3D scene creation and interaction.

Findings

Achieved state-of-the-art results on the ALFRED benchmark.

Qualitative improvements in immersion and interaction quality.

User studies confirm increased task efficiency and realism.

Abstract

Recent advances in large language models (LLMs) have significantly improved language-driven 3D content generation, but most existing approaches still treat scene generation and user interaction as separate processes, limiting the adaptability and immersive potential of interactive multimedia systems. This paper presents a unified framework that closes the loop between language-driven 3D scene generation and immersive user interaction. Given natural language instructions, the system first constructs structured scene representations using LLMs, and then optimizes spatial layouts via reinforcement learning under geometric and semantic constraints. The generated environments are deployed in a virtual reality setting to facilitate HRI-in-the-loop, where user interactions provide continuous feedback to align generated content with human perception and usability. By tightly coupling generation and interaction, the proposed framework enables more responsive, adaptive, and realistic multimedia experiences. Experiments on the ALFRED benchmark demonstrate state-of-the-art performance in task-based scene generation. Furthermore, qualitative results and user studies show consistent improvements in immersion, interaction quality, and task efficiency, highlighting the importance of closed-loop integration of generation and interaction for next-generation multimedia systems. Our project page can be found at https://proj-showcase.github.io/h3ds/.