Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches
Yongzhi Xu, Yonhon Ng, Yifu Wang, Inkyu Sa, Yunfei Duan, Zhenhong Sun, Yang Li, Pan Ji, Hongdong Li

TL;DR
This paper introduces a deep learning approach that transforms casual sketches into interactive 3D game scenes by leveraging 2D diffusion models and image understanding to guide 3D content creation.
Contribution
It presents a novel pipeline combining 2D diffusion models, image segmentation, and procedural generation to create 3D game scenes from sketches, addressing data scarcity issues.
Findings
Generates high-quality, interactive 3D scenes from sketches
Accurately follows user design intentions in scene layout
Integrates seamlessly with game engines like Unity and Unreal
Abstract
3D Content Generation is at the heart of many computer graphics applications, including video gaming, film-making, virtual and augmented reality, etc. This paper proposes a novel deep-learning based approach for automatically generating interactive and playable 3D game scenes, all from the user's casual prompts such as a hand-drawn sketch. Sketch-based input offers a natural, and convenient way to convey the user's design intention in the content creation process. To circumvent the data-deficient challenge in learning (i.e. the lack of large training data of 3D scenes), our method leverages a pre-trained 2D denoising diffusion model to generate a 2D image of the scene as the conceptual guidance. In this process, we adopt the isometric projection mode to factor out unknown camera poses while obtaining the scene layout. From the generated isometric image, we use a pre-trained image…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Motion and Animation · 3D Shape Modeling and Analysis · Computer Graphics and Visualization Techniques
MethodsDiffusion
