Contrastive Gaussian Clustering: Weakly Supervised 3D Scene Segmentation

TL;DR

This paper presents Contrastive Gaussian Clustering, a novel method for 3D scene segmentation that leverages Gaussian modeling and contrastive learning to produce view-consistent masks with high accuracy.

Contribution

It introduces a new approach combining Gaussian modeling and contrastive learning for weakly supervised 3D scene segmentation and multi-view mask consistency.

Findings

Achieves +8% IoU accuracy over state-of-the-art methods.

Capable of generating view-consistent 3D segmentation masks.

Effective even with inconsistent 2D supervision masks.

Abstract

We introduce Contrastive Gaussian Clustering, a novel approach capable of provide segmentation masks from any viewpoint and of enabling 3D segmentation of the scene. Recent works in novel-view synthesis have shown how to model the appearance of a scene via a cloud of 3D Gaussians, and how to generate accurate images from a given viewpoint by projecting on it the Gaussians before $\alpha$ blending their color. Following this example, we train a model to include also a segmentation feature vector for each Gaussian. These can then be used for 3D scene segmentation, by clustering Gaussians according to their feature vectors; and to generate 2D segmentation masks, by projecting the Gaussians on a plane and $\alpha$ blending over their segmentation features. Using a combination of contrastive learning and spatial regularization, our method can be trained on inconsistent 2D segmentation masks, and still learn to generate segmentation masks consistent across all views. Moreover, the resulting model is extremely accurate, improving the IoU accuracy of the predicted masks by $+8\%$ over the state of the art. Code and trained models will be released soon.