Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation

TL;DR

This paper introduces a factorized neural re-rendering approach for outdoor scenes that enables controllable, photorealistic photo extrapolation and 3D photo generation from Internet photo collections, addressing challenges of ambiguity and occlusion.

Contribution

It presents a novel factorized pipeline and training strategy for outdoor scene re-rendering, improving photo realism and enabling applications like photo extrapolation and 3D photo generation.

Findings

Outperforms existing methods in photo realism and scene rendering quality.

Enables controllable scene re-rendering and photo extrapolation.

Demonstrates superior results on outdoor scene datasets.

Abstract

Expanding an existing tourist photo from a partially captured scene to a full scene is one of the desired experiences for photography applications. Although photo extrapolation has been well studied, it is much more challenging to extrapolate a photo (i.e., selfie) from a narrow field of view to a wider one while maintaining a similar visual style. In this paper, we propose a factorized neural re-rendering model to produce photorealistic novel views from cluttered outdoor Internet photo collections, which enables the applications including controllable scene re-rendering, photo extrapolation and even extrapolated 3D photo generation. Specifically, we first develop a novel factorized re-rendering pipeline to handle the ambiguity in the decomposition of geometry, appearance and illumination. We also propose a composited training strategy to tackle the unexpected occlusion in Internet images. Moreover, to enhance photo-realism when extrapolating tourist photographs, we propose a novel realism augmentation process to complement appearance details, which automatically propagates the texture details from a narrow captured photo to the extrapolated neural rendered image. The experiments and photo editing examples on outdoor scenes demonstrate the superior performance of our proposed method in both photo-realism and downstream applications.