Guided Co-Modulated GAN for 360{\deg} Field of View Extrapolation

TL;DR

This paper introduces a novel guided co-modulation GAN framework for extrapolating 360-degree panoramic images from a single view, enabling high-quality, user-controlled out-painting with applications in virtual object insertion.

Contribution

It presents a new guided co-modulation approach that enhances GAN-based out-painting for panoramic images, allowing semantic control and improved visual quality.

Findings

Achieves state-of-the-art image quality in 360-degree extrapolation

Enables user-controlled semantic editing of panoramas

Facilitates photorealistic virtual object insertion

Abstract

We propose a method to extrapolate a 360{\deg} field of view from a single image that allows for user-controlled synthesis of the out-painted content. To do so, we propose improvements to an existing GAN-based in-painting architecture for out-painting panoramic image representation. Our method obtains state-of-the-art results and outperforms previous methods on standard image quality metrics. To allow controlled synthesis of out-painting, we introduce a novel guided co-modulation framework, which drives the image generation process with a common pretrained discriminative model. Doing so maintains the high visual quality of generated panoramas while enabling user-controlled semantic content in the extrapolated field of view. We demonstrate the state-of-the-art results of our method on field of view extrapolation both qualitatively and quantitatively, providing thorough analysis of our novel editing capabilities. Finally, we demonstrate that our approach benefits the photorealistic virtual insertion of highly glossy objects in photographs.