ReShader: View-Dependent Highlights for Single Image View-Synthesis

TL;DR

ReShader introduces a method to improve single-image view synthesis by separately handling pixel shading and relocation, resulting in more realistic view-dependent highlights in the generated images.

Contribution

The paper presents a novel two-step approach combining neural reshading and existing view synthesis techniques to better model view-dependent effects in single-image view synthesis.

Findings

Produces more realistic highlights in synthesized views

Effective on various real-world scenes

Outperforms previous methods in visual plausibility

Abstract

In recent years, novel view synthesis from a single image has seen significant progress thanks to the rapid advancements in 3D scene representation and image inpainting techniques. While the current approaches are able to synthesize geometrically consistent novel views, they often do not handle the view-dependent effects properly. Specifically, the highlights in their synthesized images usually appear to be glued to the surfaces, making the novel views unrealistic. To address this major problem, we make a key observation that the process of synthesizing novel views requires changing the shading of the pixels based on the novel camera, and moving them to appropriate locations. Therefore, we propose to split the view synthesis process into two independent tasks of pixel reshading and relocation. During the reshading process, we take the single image as the input and adjust its shading based on the novel camera. This reshaded image is then used as the input to an existing view synthesis method to relocate the pixels and produce the final novel view image. We propose to use a neural network to perform reshading and generate a large set of synthetic input-reshaded pairs to train our network. We demonstrate that our approach produces plausible novel view images with realistic moving highlights on a variety of real world scenes.