4K4DGen: Panoramic 4D Generation at 4K Resolution

TL;DR

This paper introduces 4K4DGen, a novel method for generating high-resolution, 360-degree panoramic 4D scenes from a single image, enabling immersive VR/AR experiences with real-time dynamic scene animation.

Contribution

It presents a pioneering pipeline that converts panoramic videos into 4K immersive 4D environments, including a panoramic denoiser and a dynamic lifting technique, addressing data scarcity and ensuring consistency.

Findings

First 4K panoramic 4D scene generation from a single image

Real-time dynamic scene animation at 4K resolution

Effective adaptation of 2D diffusion priors to panoramic 4D environments

Abstract

The blooming of virtual reality and augmented reality (VR/AR) technologies has driven an increasing demand for the creation of high-quality, immersive, and dynamic environments. However, existing generative techniques either focus solely on dynamic objects or perform outpainting from a single perspective image, failing to meet the requirements of VR/AR applications that need free-viewpoint, 360$^{\circ}$ virtual views where users can move in all directions. In this work, we tackle the challenging task of elevating a single panorama to an immersive 4D experience. For the first time, we demonstrate the capability to generate omnidirectional dynamic scenes with 360$^{\circ}$ views at 4K (4096 $\times$ 2048) resolution, thereby providing an immersive user experience. Our method introduces a pipeline that facilitates natural scene animations and optimizes a set of dynamic Gaussians using efficient splatting techniques for real-time exploration. To overcome the lack of scene-scale annotated 4D data and models, especially in panoramic formats, we propose a novel \textbf{Panoramic Denoiser} that adapts generic 2D diffusion priors to animate consistently in 360$^{\circ}$ images, transforming them into panoramic videos with dynamic scenes at targeted regions. Subsequently, we propose \textbf{Dynamic Panoramic Lifting} to elevate the panoramic video into a 4D immersive environment while preserving spatial and temporal consistency. By transferring prior knowledge from 2D models in the perspective domain to the panoramic domain and the 4D lifting with spatial appearance and geometry regularization, we achieve high-quality Panorama-to-4D generation at a resolution of 4K for the first time.