A Neural Rendering Framework for Free-Viewpoint Relighting

TL;DR

This paper introduces RNR, a neural rendering framework that explicitly models physical lighting and object properties, enabling high-quality free-viewpoint relighting and view synthesis from multi-view images.

Contribution

The paper proposes a physically based neural rendering model that explicitly incorporates environment lighting, intrinsic object attributes, and light transport, enhancing relighting and view synthesis capabilities.

Findings

RNR achieves superior relighting quality compared to existing methods.

The framework improves view synthesis by integrating physical rendering principles.

Experiments demonstrate effectiveness on both synthetic and real data.

Abstract

We present a novel Relightable Neural Renderer (RNR) for simultaneous view synthesis and relighting using multi-view image inputs. Existing neural rendering (NR) does not explicitly model the physical rendering process and hence has limited capabilities on relighting. RNR instead models image formation in terms of environment lighting, object intrinsic attributes, and light transport function (LTF), each corresponding to a learnable component. In particular, the incorporation of a physically based rendering process not only enables relighting but also improves the quality of view synthesis. Comprehensive experiments on synthetic and real data show that RNR provides a practical and effective solution for conducting free-viewpoint relighting.