Structurally Disentangled Feature Fields Distillation for 3D Understanding and Editing

TL;DR

This paper introduces a method for 3D feature understanding and editing by disentangling view-dependent and view-independent components from 2D features, enabling more precise control and manipulation of 3D objects.

Contribution

It proposes a novel approach to capture 3D features using multiple disentangled fields, improving 3D understanding and editing from 2D supervision.

Findings

Effective 3D segmentation achieved

Enhanced control over view-dependent features

Demonstrated novel editing capabilities

Abstract

Recent work has demonstrated the ability to leverage or distill pre-trained 2D features obtained using large pre-trained 2D models into 3D features, enabling impressive 3D editing and understanding capabilities using only 2D supervision. Although impressive, models assume that 3D features are captured using a single feature field and often make a simplifying assumption that features are view-independent. In this work, we propose instead to capture 3D features using multiple disentangled feature fields that capture different structural components of 3D features involving view-dependent and view-independent components, which can be learned from 2D feature supervision only. Subsequently, each element can be controlled in isolation, enabling semantic and structural understanding and editing capabilities. For instance, using a user click, one can segment 3D features corresponding to a given object and then segment, edit, or remove their view-dependent (reflective) properties. We evaluate our approach on the task of 3D segmentation and demonstrate a set of novel understanding and editing tasks.