View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields

TL;DR

This paper introduces a method to convert view-inconsistent multi-granular 2D segmentations into a hierarchical, 3D-consistent representation using an ultrametric feature space within a Neural Radiance Field, improving accuracy and consistency.

Contribution

The work presents a novel ultrametric feature space within NeRF for hierarchical 3D segmentation, addressing view inconsistency and multi-granularity challenges.

Findings

Improved accuracy over baseline methods.

Enhanced viewpoint consistency in 3D segmentation.

Qualitative success in real-world scene segmentation.

Abstract

Large-scale vision foundation models such as Segment Anything (SAM) demonstrate impressive performance in zero-shot image segmentation at multiple levels of granularity. However, these zero-shot predictions are rarely 3D-consistent. As the camera viewpoint changes in a scene, so do the segmentation predictions, as well as the characterizations of "coarse" or "fine" granularity. In this work, we address the challenging task of lifting multi-granular and view-inconsistent image segmentations into a hierarchical and 3D-consistent representation. We learn a novel feature field within a Neural Radiance Field (NeRF) representing a 3D scene, whose segmentation structure can be revealed at different scales by simply using different thresholds on feature distance. Our key idea is to learn an ultrametric feature space, which unlike a Euclidean space, exhibits transitivity in distance-based grouping, naturally leading to a hierarchical clustering. Put together, our method takes view-inconsistent multi-granularity 2D segmentations as input and produces a hierarchy of 3D-consistent segmentations as output. We evaluate our method and several baselines on synthetic datasets with multi-view images and multi-granular segmentation, showcasing improved accuracy and viewpoint-consistency. We additionally provide qualitative examples of our model's 3D hierarchical segmentations in real world scenes. The code and dataset are available at https://github.com/hardyho/ultrametric_feature_fields