Shape from Semantics: 3D Shape Generation from Multi-View Semantics

TL;DR

This paper introduces 'Shape from Semantics', a novel 3D modeling task that generates geometrically and visually coherent 3D shapes from multi-view semantic descriptions, enhancing creative 3D design.

Contribution

It proposes a new task and framework that incorporate multi-view semantics into 3D shape generation, using generative priors, LGAD for geometry completion, and physically based rendering for appearance.

Findings

Generates detailed, structured 3D meshes with coherent textures.

Ensures smooth semantic transitions across views.

Produces visually appealing 3D models with realistic geometry and appearance.

Abstract

Existing 3D reconstruction methods utilize guidances such as 2D images, 3D point clouds, shape contours and single semantics to recover the 3D surface, which limits the creative exploration of 3D modeling. In this paper, we propose a novel 3D modeling task called ``Shape from Semantics'', which aims to create 3D models whose geometry and appearance are consistent with the given text semantics when viewed from different views. The reconstructed 3D models incorporate more than one semantic elements and are easy for observers to distinguish. We adopt generative models as priors and disentangle the connection between geometry and appearance to solve this challenging problem. Specifically, we propose Local Geometry-Aware Distillation (LGAD), a strategy that employs multi-view normal-depth diffusion priors to complete partial geometries, ensuring realistic shape generation. We also integrate view-adaptive guidance scales to enable smooth semantic transitions across views. For appearance modeling, we adopt physically based rendering to generate high-quality material properties, which are subsequently baked into fabricable meshes. Extensive experimental results demonstrate that our method can generate meshes with well-structured, intricately detailed geometries, coherent textures, and smooth transitions, resulting in visually appealing 3D shape designs. Project page: https://shapefromsemantics.github.io