IReNe: Instant Recoloring of Neural Radiance Fields

TL;DR

IReNe enables fast, precise, and view-consistent color editing of neural radiance fields by fine-tuning only specific network layers, significantly improving editing speed and accuracy.

Contribution

This paper introduces IReNe, a novel method for rapid, boundary-aware color editing of NeRFs using minimal retraining and neuron classification techniques.

Findings

Achieves 5x to 500x faster editing speeds than competitors.

Maintains high photorealism and multi-view consistency after edits.

Effectively controls object boundaries through a trainable segmentation module.

Abstract

Advances in NERFs have allowed for 3D scene reconstructions and novel view synthesis. Yet, efficiently editing these representations while retaining photorealism is an emerging challenge. Recent methods face three primary limitations: they're slow for interactive use, lack precision at object boundaries, and struggle to ensure multi-view consistency. We introduce IReNe to address these limitations, enabling swift, near real-time color editing in NeRF. Leveraging a pre-trained NeRF model and a single training image with user-applied color edits, IReNe swiftly adjusts network parameters in seconds. This adjustment allows the model to generate new scene views, accurately representing the color changes from the training image while also controlling object boundaries and view-specific effects. Object boundary control is achieved by integrating a trainable segmentation module into the model. The process gains efficiency by retraining only the weights of the last network layer. We observed that neurons in this layer can be classified into those responsible for view-dependent appearance and those contributing to diffuse appearance. We introduce an automated classification approach to identify these neuron types and exclusively fine-tune the weights of the diffuse neurons. This further accelerates training and ensures consistent color edits across different views. A thorough validation on a new dataset, with edited object colors, shows significant quantitative and qualitative advancements over competitors, accelerating speeds by 5x to 500x.