Geometry Aware Field-to-field Transformations for 3D Semantic Segmentation

TL;DR

This paper introduces a novel method for 3D semantic segmentation that uses 2D supervision and neural radiance fields to efficiently learn scene representations, enabling few-shot segmentation across various scene types.

Contribution

It proposes a geometry-aware, NeRF-based feature extraction approach for 3D segmentation that operates with minimal supervision and is scene-agnostic.

Findings

Achieves sample-efficient 3D segmentation from 2D data

Enables few-shot segmentation through unsupervised feature learning

Works with any scene parameterization fitted with NeRF

Abstract

We present a novel approach to perform 3D semantic segmentation solely from 2D supervision by leveraging Neural Radiance Fields (NeRFs). By extracting features along a surface point cloud, we achieve a compact representation of the scene which is sample-efficient and conducive to 3D reasoning. Learning this feature space in an unsupervised manner via masked autoencoding enables few-shot segmentation. Our method is agnostic to the scene parameterization, working on scenes fit with any type of NeRF.