Orthogonal Spatial-temporal Distributional Transfer for 4D Generation

TL;DR

This paper introduces a novel framework for 4D content generation that leverages spatial priors from 3D models and temporal priors from video models, overcoming dataset limitations to produce high-quality, consistent 4D synthesis.

Contribution

It proposes a new orthogonal transfer mechanism and a disentangled diffusion model to enhance 4D synthesis by effectively transferring spatial and temporal features.

Findings

Outperforms existing methods in 4D synthesis quality

Achieves superior spatial-temporal consistency

Demonstrates effective feature transfer across domains

Abstract

In the AIGC era, generating high-quality 4D content has garnered increasing research attention. Unfortunately, current 4D synthesis research is severely constrained by the lack of large-scale 4D datasets, preventing models from adequately learning the critical spatial-temporal features necessary for high-quality 4D generation, thus hindering progress in this domain. To combat this, we propose a novel framework that transfers rich spatial priors from existing 3D diffusion models and temporal priors from video diffusion models to enhance 4D synthesis. We develop a spatial-temporal-disentangled 4D (STD-4D) Diffusion model, which synthesizes 4D-aware videos through disentangled spatial and temporal latents. To facilitate the best feature transfer, we design a novel Orthogonal Spatial-temporal Distributional Transfer (Orster) mechanism, where the spatiotemporal feature distributions are carefully modeled and injected into the STD-4D Diffusion. Furthermore, during the 4D construction, we devise a spatial-temporal-aware HexPlane (ST-HexPlane) to integrate the transferred spatiotemporal features, thereby improving 4D deformation and 4D Gaussian feature modeling. Experiments demonstrate that our method significantly outperforms existing approaches, achieving superior spatial-temporal consistency and higher-quality 4D synthesis.