LaTeRF: Label and Text Driven Object Radiance Fields

TL;DR

LaTeRF is a novel method that extracts 3D object representations from scenes using weak supervision, natural language, and point-labels, leveraging CLIP and NeRF extensions for high-fidelity results.

Contribution

It introduces a new approach combining language, point-labels, and NeRF extensions to improve 3D object extraction from scenes with minimal supervision.

Findings

High-fidelity object extraction demonstrated on synthetic datasets.

Effective occlusion inpainting using CLIP's latent space.

Extensive ablation studies validate design choices.

Abstract

Obtaining 3D object representations is important for creating photo-realistic simulations and for collecting AR and VR assets. Neural fields have shown their effectiveness in learning a continuous volumetric representation of a scene from 2D images, but acquiring object representations from these models with weak supervision remains an open challenge. In this paper we introduce LaTeRF, a method for extracting an object of interest from a scene given 2D images of the entire scene, known camera poses, a natural language description of the object, and a set of point-labels of object and non-object points in the input images. To faithfully extract the object from the scene, LaTeRF extends the NeRF formulation with an additional `objectness' probability at each 3D point. Additionally, we leverage the rich latent space of a pre-trained CLIP model combined with our differentiable object renderer, to inpaint the occluded parts of the object. We demonstrate high-fidelity object extraction on both synthetic and real-world datasets and justify our design choices through an extensive ablation study.