360-Degree Panorama Generation from Few Unregistered NFoV Images

TL;DR

This paper introduces PanoDiff, a novel pipeline that generates high-quality 360-degree panoramas from few unregistered NFoV images using a two-stage angle prediction and a diffusion-based model with control signals.

Contribution

The paper presents a new method combining angle prediction and diffusion models to generate controllable panoramas from arbitrary NFoV images, improving quality and flexibility.

Findings

Achieves state-of-the-art panorama generation quality

Provides high controllability with control signals

Handles various numbers of NFoV inputs effectively

Abstract

360$^\circ$ panoramas are extensively utilized as environmental light sources in computer graphics. However, capturing a 360$^\circ$ $\times$ 180$^\circ$ panorama poses challenges due to the necessity of specialized and costly equipment, and additional human resources. Prior studies develop various learning-based generative methods to synthesize panoramas from a single Narrow Field-of-View (NFoV) image, but they are limited in alterable input patterns, generation quality, and controllability. To address these issues, we propose a novel pipeline called PanoDiff, which efficiently generates complete 360$^\circ$ panoramas using one or more unregistered NFoV images captured from arbitrary angles. Our approach has two primary components to overcome the limitations. Firstly, a two-stage angle prediction module to handle various numbers of NFoV inputs. Secondly, a novel latent diffusion-based panorama generation model uses incomplete panorama and text prompts as control signals and utilizes several geometric augmentation schemes to ensure geometric properties in generated panoramas. Experiments show that PanoDiff achieves state-of-the-art panoramic generation quality and high controllability, making it suitable for applications such as content editing.