Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs

TL;DR

Dreamcrafter is a VR system that enables immersive, real-time editing of 3D radiance fields using generative AI, combining direct manipulation and natural language control to enhance scene creation and editing.

Contribution

It introduces a modular VR editing system that integrates generative AI with proxy representations for real-time 3D scene editing, addressing latency and control issues.

Findings

Control preferences vary among users

Generative AI interfaces boost creativity

Proxy representations enable interaction during high-latency operations

Abstract

Authoring 3D scenes is a central task for spatial computing applications. Competing visions for lowering existing barriers are (1) focus on immersive, direct manipulation of 3D content or (2) leverage AI techniques that capture real scenes (3D Radiance Fields such as, NeRFs, 3D Gaussian Splatting) and modify them at a higher level of abstraction, at the cost of high latency. We unify the complementary strengths of these approaches and investigate how to integrate generative AI advances into real-time, immersive 3D Radiance Field editing. We introduce Dreamcrafter, a VR-based 3D scene editing system that: (1) provides a modular architecture to integrate generative AI algorithms; (2) combines different levels of control for creating objects, including natural language and direct manipulation; and (3) introduces proxy representations that support interaction during high-latency operations. We contribute empirical findings on control preferences and discuss how generative AI interfaces beyond text input enhance creativity in scene editing and world building.