CoNeRF: Controllable Neural Radiance Fields

TL;DR

CoNeRF introduces a method to control and manipulate neural radiance fields with minimal annotations, enabling intuitive scene editing and attribute control from limited data, including novel view and attribute re-rendering from a single video.

Contribution

The paper presents a novel framework that allows controllable neural radiance fields with automatic attribute discovery and minimal supervision, extending 3D scene representations beyond view synthesis.

Findings

Enables user-controlled scene editing with few annotations

Achieves novel view and attribute re-rendering from a single video

Demonstrates controllable scene manipulation across various scenes

Abstract

We extend neural 3D representations to allow for intuitive and interpretable user control beyond novel view rendering (i.e. camera control). We allow the user to annotate which part of the scene one wishes to control with just a small number of mask annotations in the training images. Our key idea is to treat the attributes as latent variables that are regressed by the neural network given the scene encoding. This leads to a few-shot learning framework, where attributes are discovered automatically by the framework, when annotations are not provided. We apply our method to various scenes with different types of controllable attributes (e.g. expression control on human faces, or state control in movement of inanimate objects). Overall, we demonstrate, to the best of our knowledge, for the first time novel view and novel attribute re-rendering of scenes from a single video.