ViewPoint: Panoramic Video Generation with Pretrained Diffusion Models

TL;DR

This paper introduces ViewPoint, a framework that leverages pretrained perspective video models and a novel panorama representation to generate high-quality, spatially consistent panoramic videos, advancing the state-of-the-art in VR content creation.

Contribution

The paper proposes a new panorama representation called ViewPoint map and a Pano-Perspective attention mechanism to effectively utilize pretrained perspective models for panoramic video synthesis.

Findings

Achieves state-of-the-art performance in panoramic video generation.

Produces highly dynamic and spatially consistent videos.

Outperforms previous methods in quality and realism.

Abstract

Panoramic video generation aims to synthesize 360-degree immersive videos, holding significant importance in the fields of VR, world models, and spatial intelligence. Existing works fail to synthesize high-quality panoramic videos due to the inherent modality gap between panoramic data and perspective data, which constitutes the majority of the training data for modern diffusion models. In this paper, we propose a novel framework utilizing pretrained perspective video models for generating panoramic videos. Specifically, we design a novel panorama representation named ViewPoint map, which possesses global spatial continuity and fine-grained visual details simultaneously. With our proposed Pano-Perspective attention mechanism, the model benefits from pretrained perspective priors and captures the panoramic spatial correlations of the ViewPoint map effectively. Extensive experiments demonstrate that our method can synthesize highly dynamic and spatially consistent panoramic videos, achieving state-of-the-art performance and surpassing previous methods.