Appearance Editing with Free-viewpoint Neural Rendering

TL;DR

This paper introduces a neural rendering method that enables simultaneous view synthesis and appearance editing of scenes from multi-view images, by explicitly disentangling appearance and lighting representations.

Contribution

It proposes a novel framework that learns independent lighting representations and BRDF estimates, allowing flexible appearance editing alongside high-quality view synthesis.

Findings

Achieves view synthesis performance comparable to state-of-the-art methods.

Demonstrates plausible appearance editing on real scenes.

Generalizes to arbitrary appearance changes while maintaining view synthesis quality.

Abstract

We present a neural rendering framework for simultaneous view synthesis and appearance editing of a scene from multi-view images captured under known environment illumination. Existing approaches either achieve view synthesis alone or view synthesis along with relighting, without direct control over the scene's appearance. Our approach explicitly disentangles the appearance and learns a lighting representation that is independent of it. Specifically, we independently estimate the BRDF and use it to learn a lighting-only representation of the scene. Such disentanglement allows our approach to generalize to arbitrary changes in appearance while performing view synthesis. We show results of editing the appearance of a real scene, demonstrating that our approach produces plausible appearance editing. The performance of our view synthesis approach is demonstrated to be at par with state-of-the-art approaches on both real and synthetic data.