Ghost on the Shell: An Expressive Representation of General 3D Shapes
Zhen Liu, Yao Feng, Yuliang Xiu, Weiyang Liu, Liam Paull, Michael J., Black, Bernhard Sch\"olkopf
TL;DR
This paper introduces G-Shell, a novel differentiable 3D shape representation that effectively models both watertight and open, non-watertight meshes, enabling high-quality reconstruction and generative modeling.
Contribution
The paper proposes G-Shell, a new manifold signed distance field-based representation that handles arbitrary topology, including open surfaces, for 3D shape reconstruction and generation.
Findings
Achieves state-of-the-art results in non-watertight mesh reconstruction.
Effectively models both watertight and open 3D shapes.
Supports differentiable rasterization for multiview reconstruction.
Abstract
The creation of photorealistic virtual worlds requires the accurate modeling of 3D surface geometry for a wide range of objects. For this, meshes are appealing since they 1) enable fast physics-based rendering with realistic material and lighting, 2) support physical simulation, and 3) are memory-efficient for modern graphics pipelines. Recent work on reconstructing and statistically modeling 3D shape, however, has critiqued meshes as being topologically inflexible. To capture a wide range of object shapes, any 3D representation must be able to model solid, watertight, shapes as well as thin, open, surfaces. Recent work has focused on the former, and methods for reconstructing open surfaces do not support fast reconstruction with material and lighting or unconditional generative modelling. Inspired by the observation that open surfaces can be seen as islands floating on watertight…
Peer Reviews
Decision·ICLR 2024 oral
1. The topic studied in this paper is interesting and important. 2. Being able to model thin, open surfaces will be useful for a couple of applications. 3. The proposed method is technically sound. 4. Experiments show that the proposed method is effective in modeling non-watertight meshes.
1. While the proposed method is faster than other methods as shown in Table 3, 3 hours is still too long. 2. The thin shape examples in Figure 4 and Figure 7 don't have complicated geometry. If there is a more complicated geometry, how well would the proposed method perform in reconstructing / modeling? For instance, when dropping a cloth onto an object, the cloth will have a lot of folds, wrinkles and even a lot of self-contacts. Can the proposed method deal with this case?
The basic idea is elegant, and it avoids the problems of unsigned distance fields. The comparisons to previous work are also compelling. The generative modeling results look cool, though it would be nice to see some examples other than clothing if other datasets are available.
The authors claim this is the first work to propose a differentiable representation suitable for both open and closed surfaces. Though they mention representations for open surfaces based on unsigned distance fields, they should also include citations to the following two works, which offer alternative approaches: - D. Palmer, D. Smirnov, S. Wang, A. Chern, and J. Solomon, “DeepCurrents: learning implicit representations of shapes with boundaries,” in Proceedings of the IEEE/CVF Conference on Co
1. A nice extension of 3D grid representation that can handle open surface meshes. 2. Impressive experimental results on open surface reconstruction and generation.
While the experimental results are impressive, it is not clear whether the proposed modified marching cube or tetrahedra with mSDF value can guarantee the correct topology of the 3D mesh. For example, an isolated edge with no incident triangles.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsComputer Graphics and Visualization Techniques · 3D Shape Modeling and Analysis · Advanced Vision and Imaging