Mixed-View Panorama Synthesis using Geospatially Guided Diffusion

TL;DR

This paper presents a novel approach for mixed-view panorama synthesis that combines input panoramas and satellite images using diffusion models and attention mechanisms, enabling realistic panorama generation even with sparse data.

Contribution

It introduces the task of mixed-view panorama synthesis and proposes a diffusion-based, attention-augmented model to synthesize panoramas from limited and diverse input data.

Findings

Effective synthesis with sparse panoramas

Handles diverse geographic locations

Outperforms previous methods in realism and robustness

Abstract

We introduce the task of mixed-view panorama synthesis, where the goal is to synthesize a novel panorama given a small set of input panoramas and a satellite image of the area. This contrasts with previous work which only uses input panoramas (same-view synthesis), or an input satellite image (cross-view synthesis). We argue that the mixed-view setting is the most natural to support panorama synthesis for arbitrary locations worldwide. A critical challenge is that the spatial coverage of panoramas is uneven, with few panoramas available in many regions of the world. We introduce an approach that utilizes diffusion-based modeling and an attention-based architecture for extracting information from all available input imagery. Experimental results demonstrate the effectiveness of our proposed method. In particular, our model can handle scenarios when the available panoramas are sparse or far from the location of the panorama we are attempting to synthesize. The project page is available at https://mixed-view.github.io