MiXR: Harvesting and Recomposing Geometry from Real-World Objects for In-Situ 3D Design

TL;DR

MiXR is an XR system that allows users to harvest geometry from real-world objects and assemble new 3D models with generative AI, enabling precise spatial control in in-situ 3D design.

Contribution

The paper introduces MiXR, a hybrid system combining direct manipulation and AI synthesis for in-situ 3D modeling from real-world geometry.

Findings

Participants achieved more accurate designs with MiXR.

Users felt more in control and experienced lower cognitive workload.

MiXR outperformed a generative composition baseline in user study.

Abstract

Recent developments in 3D generative AI enable users to create bespoke 3D models from text or image prompts. However, these approaches provide limited control over spatial structure, making them ill suited for tasks requiring precise geometric composition. We present MiXR, an XR system for in-situ compositional modeling that enables users to create new 3D models by harvesting geometry from their environment. Users extract segments from captured objects and assemble new artifacts through direct 3D manipulation, while generative AI synthesizes a coherent model from the user-defined composition. This hybrid workflow allows users to define spatial structure explicitly while delegating geometric refinement to generative models, enabling them to specify spatial intent that is difficult to express through verbal prompts alone. In a controlled user study ($N=12$), participants using MiXR rated their designs as significantly closer to the target, felt more in control, and experienced lower cognitive workload compared to a generative composition baseline.