4DSTR: Advancing Generative 4D Gaussians with Spatial-Temporal Rectification for High-Quality and Consistent 4D Generation

TL;DR

This paper introduces 4DSTR, a novel generative model for 4D content that ensures high-quality, consistent, and adaptable dynamic 4D generation by leveraging spatial-temporal rectification and adaptive strategies.

Contribution

The paper proposes a new 4D generation network, 4DSTR, which enhances temporal consistency and adapts to rapid changes using spatial-temporal rectification and dynamic Gaussian point management.

Findings

Achieves state-of-the-art results in video-to-4D generation.

Improves spatial-temporal consistency in 4D content.

Effectively handles rapid temporal variations.

Abstract

Remarkable advances in recent 2D image and 3D shape generation have induced a significant focus on dynamic 4D content generation. However, previous 4D generation methods commonly struggle to maintain spatial-temporal consistency and adapt poorly to rapid temporal variations, due to the lack of effective spatial-temporal modeling. To address these problems, we propose a novel 4D generation network called 4DSTR, which modulates generative 4D Gaussian Splatting with spatial-temporal rectification. Specifically, temporal correlation across generated 4D sequences is designed to rectify deformable scales and rotations and guarantee temporal consistency. Furthermore, an adaptive spatial densification and pruning strategy is proposed to address significant temporal variations by dynamically adding or deleting Gaussian points with the awareness of their pre-frame movements. Extensive experiments demonstrate that our 4DSTR achieves state-of-the-art performance in video-to-4D generation, excelling in reconstruction quality, spatial-temporal consistency, and adaptation to rapid temporal movements.