Mani-GS: Gaussian Splatting Manipulation with Triangular Mesh
Xiangjun Gao, Xiaoyu Li, Yiyu Zhuang, Qi Zhang, Wenbo Hu, Chaopeng, Zhang, Yao Yao, Ying Shan, Long Quan
TL;DR
This paper introduces Mani-GS, a method that uses triangular meshes to manipulate 3D Gaussian Splatting representations, enabling controllable, high-fidelity editing and deformation with faster rendering compared to previous neural radiance field methods.
Contribution
The paper presents a novel approach for manipulating 3D Gaussian Splatting using a triangle shape-aware binding, allowing flexible editing while maintaining high rendering quality.
Findings
Effective handling of large deformations and local manipulations.
Preserves high-fidelity rendering after manipulation.
Outperforms baseline approaches in experiments.
Abstract
Neural 3D representations such as Neural Radiance Fields (NeRF), excel at producing photo-realistic rendering results but lack the flexibility for manipulation and editing which is crucial for content creation. Previous works have attempted to address this issue by deforming a NeRF in canonical space or manipulating the radiance field based on an explicit mesh. However, manipulating NeRF is not highly controllable and requires a long training and inference time. With the emergence of 3D Gaussian Splatting (3DGS), extremely high-fidelity novel view synthesis can be achieved using an explicit point-based 3D representation with much faster training and rendering speed. However, there is still a lack of effective means to manipulate 3DGS freely while maintaining rendering quality. In this work, we aim to tackle the challenge of achieving manipulable photo-realistic rendering. We propose to…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
The strengths of this work rely mainly on addressing a highly relevant problem and achieving great values compared to their chosen state-of-the-art methods.
This work suffers from a few larger issues: - Poor writing quality. Here, we mostly mean that the paper heavily introduces and talks about NeRF in the introduction and related work, while this is not relevant for understanding the paper. Further, structurally the paper needs some improvements (for example Figure is mentioned on page 4 but not seen until page 6) - In general, while the method works decently, the contributions do not seem to be enough - Compared to SuGaR, the author here uses bet
1. The paper is well written and straightforward. 2. The paper proposes to bind Gaussians to a local triangle space, which maintains the local rigidity and preserves the relative location between Gaussians, allowing the method to preserve the high-fidelity rendering results after manipulation. 2. The manipulation results are vivid and interesting, especially the soft body simulation. 3. The authors demonstrate the editability of the method on three different tasks, which shows the capability of
1. The authors need to compare their method with GaMeS or Mesh-GS to demonstrate their contributions. In comparison to GaMeS which constrains the Gaussians on the surface exactly, the main contributions of Mani-GS are (1) attaching the Guassians to local space rather than global space, and (2) allowing Gaussians to offset out of the attached triangle. Could the authors provide some qualitative results that support those two contributions? In terms of the rendering quality given an inaccurate mes
- The paper is well written, and the analysis is comprehensive. - The idea of correlating the scale of 3D Gaussians with the shape of the triangles to better handle large-scale deformations is reasonable. - The experimental results appear to be valid.
- 3D Gaussian Spatting achieves high-quality rendering results primarily due to its split/clone mechanism, which adaptively adjusts the number of points in the scene. However, this paper limits the number of Gaussians in each triangle face, which may restrict its fitting capability. Nevertheless, the rendering metrics in Table 1 appear to be very high, with some even exceeding those of the original 3DGS; this raises questions. - The main innovation of this paper lies in the introduction of $e$ i
The paper is reasonably well written and easy to understand. It addresses the important challenge of editing 3D scenes represented by 3D Gaussians. The qualitative results look compelling and quantitatively outperform the sugar baseline.
There are several weaknesses: 1. The proposed methods is incremental compared with sugar. The proposed method is basically sugar, which binds the optimized 3D Gaussians to the mesh surface. with an additional offset. This seems like a simple extension. More advanced extensions of sugar already exists, including Guedon and Lepetit, Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Rendering, ECCV 2024 which model a much broader class of objects than both sugar and the propose
1. The article has a clear logic and provides an in-depth analysis of the problem. For example, when discussing how to bind Gaussians to the mesh (Sec.3.3), the authors compared two alternative methods ("Gaussians on Mesh" and "Gaussians on Mesh with Offset"), analyzing the principles, advantages, and disadvantages of each. Another example is the authors' discussion of the results from different mesh extraction methods (Sec.3.2). 2. The supplementary materials are meticulously prepared, and the
1. The discussion of some works is insufficient. For example, GaMeS and Mesh-GS are mentioned in the related work section, but as the most closely related and recent Gaussian methods, they are not included in the experimental comparisons. Methodologically, I feel that the Gaussian binding approach in this article is very similar to that of GaMeS, yet the authors do not discuss this point. The baselines the authors compare are outdated and are insufficient to demonstrate the superiority of their
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsImage and Object Detection Techniques · Image Processing and 3D Reconstruction · Industrial Vision Systems and Defect Detection