Self-supervised Learning of Hybrid Part-aware 3D Representations of 2D Gaussians and Superquadrics

TL;DR

This paper presents PartGS, a self-supervised framework that combines 2D Gaussians and superquadrics to decompose 3D objects into meaningful parts, improving interpretability and reconstruction quality.

Contribution

It introduces a novel hybrid, part-aware 3D representation method that jointly optimizes superquadrics and 2D Gaussians using multi-view images, enhancing interpretability and detail in 3D reconstructions.

Findings

Outperforms state-of-the-art methods on DTU, ShapeNet, and real-world datasets.

Provides meaningful part decompositions of 3D objects.

Achieves high-fidelity 3D reconstructions with detailed textures and shapes.

Abstract

Low-level 3D representations, such as point clouds, meshes, NeRFs and 3D Gaussians, are commonly used for modeling 3D objects and scenes. However, cognitive studies indicate that human perception operates at higher levels and interprets 3D environments by decomposing them into meaningful structural parts, rather than low-level elements like points or voxels. Structured geometric decomposition enhances scene interpretability and facilitates downstream tasks requiring component-level manipulation. In this work, we introduce PartGS, a self-supervised part-aware reconstruction framework that integrates 2D Gaussians and superquadrics to parse objects and scenes into an interpretable decomposition, leveraging multi-view image inputs to uncover 3D structural information. Our method jointly optimizes superquadric meshes and Gaussians by coupling their parameters within a hybrid representation. On one hand, superquadrics enable the representation of a wide range of shape primitives, facilitating flexible and meaningful decompositions. On the other hand, 2D Gaussians capture detailed texture and geometric details, ensuring high-fidelity appearance and geometry reconstruction. Operating in a self-supervised manner, our approach demonstrates superior performance compared to state-of-the-art methods across extensive experiments on the DTU, ShapeNet, and real-world datasets.