CinemaWorld: Generative Augmented Reality with LLMs and 3D Scene Generation for Movie Augmentation

TL;DR

CinemaWorld is a novel augmented reality system that uses large language models and generative AI to create synchronized 3D content from movies, enhancing immersion during film viewing.

Contribution

It introduces a new AR system that automatically generates and embeds 3D movie augmentations in real-world environments, combining multimodal LLMs and generative AI techniques.

Findings

Enhanced viewer immersion and enjoyment

Effective synchronization of 3D content with film scenes

Positive feedback from users and film creators

Abstract

We introduce CinemaWorld, a generative augmented reality system that augments the viewer's physical surroundings with automatically generated mixed reality 3D content extracted from and synchronized with 2D movie scenes. Our system preprocesses films to extract key features using multimodal large language models (LLMs), generates dynamic 3D augmentations with generative AI, and embeds them spatially into the viewer's physical environment on the Meta Quest 3. To explore the design space of CinemaWorld, we conducted an elicitation study with eight film students, which led us to identify several key augmentation types, including particle effects, surrounding objects, textural overlays, character-driven augmentation, and lighting effects. We evaluated our system through a technical evaluation (N=100 video clips), a user study (N=12), and expert interviews with film creators (N=8). Results indicate that CinemaWorld enhances immersion and enjoyment, suggesting its potential to enrich the film-viewing experience.