Hallucinating 360{\deg}: Panoramic Street-View Generation via Local Scenes Diffusion and Probabilistic Prompting

TL;DR

This paper introduces Percep360, a novel panoramic street-view generation method for autonomous driving, utilizing local scene diffusion and probabilistic prompting to produce coherent, controllable, and high-quality panoramic images that improve perception tasks.

Contribution

Percep360 is the first panoramic generation approach that leverages stitched images as supervision, combining diffusion and probabilistic prompting for coherence and controllability.

Findings

Generated images outperform stitched images in quality metrics.

Enhanced downstream BEV segmentation performance.

Effective controllability of panoramic image generation.

Abstract

Panoramic perception holds significant potential for autonomous driving, enabling vehicles to acquire a comprehensive 360{\deg} surround view in a single shot. However, autonomous driving is a data-driven task. Complete panoramic data acquisition requires complex sampling systems and annotation pipelines, which are time-consuming and labor-intensive. Although existing street view generation models have demonstrated strong data regeneration capabilities, they can only learn from the fixed data distribution of existing datasets and cannot leverage stitched pinhole images as a supervisory signal. In this paper, we propose the first panoramic generation method Percep360 for autonomous driving. Percep360 enables coherent generation of panoramic data with control signals based on the stitched panoramic data. Percep360 focuses on two key aspects: coherence and controllability. Specifically, to overcome the inherent information loss caused by the pinhole sampling process, we propose the Local Scenes Diffusion Method (LSDM). LSDM reformulates the panorama generation as a spatially continuous diffusion process, bridging the gaps between different data distributions. Additionally, to achieve the controllable generation of panoramic images, we propose a Probabilistic Prompting Method (PPM). PPM dynamically selects the most relevant control cues, enabling controllable panoramic image generation. We evaluate the effectiveness of the generated images from three perspectives: image quality assessment (i.e., no-reference and with reference), controllability, and their utility in real-world Bird's Eye View (BEV) segmentation. Notably, the generated data consistently outperforms the original stitched images in no-reference quality metrics and enhances downstream perception models. The source code will be publicly available at https://github.com/FeiT-FeiTeng/Percep360.