Leveraging BEV Paradigm for Ground-to-Aerial Image Synthesis

TL;DR

SkyDiffusion introduces a novel diffusion-based approach utilizing the Bird's-Eye View paradigm for realistic ground-to-aerial image synthesis, effectively bridging domain gaps and handling occlusions in dense urban scenes.

Contribution

The paper proposes SkyDiffusion, a new method combining Curved-BEV and BEV-guided diffusion models, along with a new dataset for diverse ground-to-aerial image synthesis tasks.

Findings

Outperforms state-of-the-art methods on multiple datasets

Generates realistic, content-consistent aerial images

Effective in dense urban and diverse scenarios

Abstract

Ground-to-aerial image synthesis focuses on generating realistic aerial images from corresponding ground street view images while maintaining consistent content layout, simulating a top-down view. The significant viewpoint difference leads to domain gaps between views, and dense urban scenes limit the visible range of street views, making this cross-view generation task particularly challenging. In this paper, we introduce SkyDiffusion, a novel cross-view generation method for synthesizing aerial images from street view images, utilizing a diffusion model and the Bird's-Eye View (BEV) paradigm. The Curved-BEV method in SkyDiffusion converts street-view images into a BEV perspective, effectively bridging the domain gap, and employs a "multi-to-one" mapping strategy to address occlusion issues in dense urban scenes. Next, SkyDiffusion designed a BEV-guided diffusion model to generate content-consistent and realistic aerial images. Additionally, we introduce a novel dataset, Ground2Aerial-3, designed for diverse ground-to-aerial image synthesis applications, including disaster scene aerial synthesis, low-altitude UAV image synthesis, and historical high-resolution satellite image synthesis tasks. Experimental results demonstrate that SkyDiffusion outperforms state-of-the-art methods on cross-view datasets across natural (CVUSA), suburban (CVACT), urban (VIGOR-Chicago), and various application scenarios (G2A-3), achieving realistic and content-consistent aerial image generation. The code, datasets and more information of this work can be found at https://opendatalab.github.io/skydiffusion/ .