PivotMesh: Generic 3D Mesh Generation via Pivot Vertices Guidance
Haohan Weng, Yikai Wang, Tong Zhang, C. L. Philip Chen, Jun Zhu
TL;DR
PivotMesh introduces a scalable, transformer-based framework for generating detailed 3D meshes by hierarchically modeling pivot vertices and mesh tokens, enabling effective learning from both small and large datasets.
Contribution
The paper proposes PivotMesh, a novel hierarchical mesh generation method that models pivot vertices and mesh tokens, improving scalability and controllability over existing native mesh generation approaches.
Findings
Effective learning from small and large datasets like Shapenet and Objaverse.
Generates compact, sharp 3D meshes across various categories.
Demonstrates versatility and potential for native mesh modeling.
Abstract
Generating compact and sharply detailed 3D meshes poses a significant challenge for current 3D generative models. Different from extracting dense meshes from neural representation, some recent works try to model the native mesh distribution (i.e., a set of triangles), which generates more compact results as humans crafted. However, due to the complexity and variety of mesh topology, these methods are typically limited to small datasets with specific categories and are hard to extend. In this paper, we introduce a generic and scalable mesh generation framework PivotMesh, which makes an initial attempt to extend the native mesh generation to large-scale datasets. We employ a transformer-based auto-encoder to encode meshes into discrete tokens and decode them from face level to vertex level hierarchically. Subsequently, to model the complex typology, we first learn to generate pivot…
Peer Reviews
Decision·ICLR 2025 Poster
1. PivotMesh addresses a significant challenge in 3D mesh generation by proposing a scalable framework that can handle large-scale datasets with simplified triangles. The use of pivot vertices as a coarse representation for guiding mesh generation is innovative and effectively handles complex topologies. 2. The paper provides a thorough evaluation of PivotMesh across various datasets and applications, including mesh generation, variation, and refinement. The generated meshes are of high quality,
1. Diversity of Generated Meshes: While the paper shows diverse mesh generation, a more systematic analysis or quantification of diversity could strengthen the results. The evaluations on more diverse shapes with complex topologies are encouraged to be conducted, such as some shapes with lots of holes, the thin structures (ficus in nerf-synthetic data), and so on. 2. More direct comparisons with the current state-of-the-art methods, especially in terms of computational efficiency and mesh qualit
- The exploration of generating pivot vertices before faces appears to be a novel approach. Ablation studies validate its effectiveness, particularly without pivot guidance and degrees selection. - Extensive experiments compare unconditional generation quality across both small and large datasets, such as ShapeNet and ObjaverseXL. The proposed framework outperforms existing methods in all settings. - The framework demonstrates versatility through various applications, including point cloud-guide
Although concurrent works are mentioned, several studies have begun exploring the generation of compact meshes in large datasets, demonstrating some success to a certain extent. It is recommended to revise the introduction and abstract to avoid claiming that existing work is difficult to extend to large datasets.
- A transformer-based auto-encoder is proposed to encode the triangle meshes into discrete tokens. - An auto-regressive transformer is proposed to generate the complete mesh tokens from pivot vertices. - By conditioning on pivot vertices, the proposed method outperforms PolyGen and MeshGPT. - Downstream applications of the method are shown for mesh variation and coarse mesh refinement.
- Both the auto-encoder and auto-regressive transformer use 24 layers of transformers, resulting the method to be memory taken and less efficient.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
Topics3D Shape Modeling and Analysis · Image Processing and 3D Reconstruction · Computer Graphics and Visualization Techniques
MethodsAttention Is All You Need · Sparse Evolutionary Training · Linear Layer · Byte Pair Encoding · Label Smoothing · Adam · Residual Connection · Position-Wise Feed-Forward Layer · Multi-Head Attention · Dropout