MS2Mesh-XR: Multi-modal Sketch-to-Mesh Generation in XR Environments

TL;DR

MS2Mesh-XR introduces a fast, multi-modal sketch-to-3D mesh pipeline for XR environments, combining hand-drawn sketches and voice inputs with AI to enable real-time object creation and manipulation.

Contribution

The paper presents a novel multi-modal pipeline integrating sketch, voice, and AI for rapid 3D object generation in XR, improving user interaction and creation speed.

Findings

Generates high-quality 3D meshes in under 20 seconds.

Enables intuitive sketch and voice-based object creation in XR.

Demonstrates practical XR use cases with immersive visualization.

Abstract

We present MS2Mesh-XR, a novel multi-modal sketch-to-mesh generation pipeline that enables users to create realistic 3D objects in extended reality (XR) environments using hand-drawn sketches assisted by voice inputs. In specific, users can intuitively sketch objects using natural hand movements in mid-air within a virtual environment. By integrating voice inputs, we devise ControlNet to infer realistic images based on the drawn sketches and interpreted text prompts. Users can then review and select their preferred image, which is subsequently reconstructed into a detailed 3D mesh using the Convolutional Reconstruction Model. In particular, our proposed pipeline can generate a high-quality 3D mesh in less than 20 seconds, allowing for immersive visualization and manipulation in run-time XR scenes. We demonstrate the practicability of our pipeline through two use cases in XR settings. By leveraging natural user inputs and cutting-edge generative AI capabilities, our approach can significantly facilitate XR-based creative production and enhance user experiences. Our code and demo will be available at: https://yueqiu0911.github.io/MS2Mesh-XR/