Not All Frame Features Are Equal: Video-to-4D Generation via Decoupling Dynamic-Static Features

TL;DR

This paper introduces DS4D, a novel method for video-to-4D generation that decouples dynamic and static features to improve dynamic representation and reduce overfitting, achieving state-of-the-art results.

Contribution

It proposes a dynamic-static feature decoupling module and a temporal-spatial similarity fusion module to enhance dynamic features in video-to-4D generation.

Findings

Achieves state-of-the-art results in video-to-4D tasks.

Effectively handles dynamic regions in videos, reducing static region overfitting.

Demonstrates effectiveness on real-world 4D scene datasets.

Abstract

Recently, the generation of dynamic 3D objects from a video has shown impressive results. Existing methods directly optimize Gaussians using whole information in frames. However, when dynamic regions are interwoven with static regions within frames, particularly if the static regions account for a large proportion, existing methods often overlook information in dynamic regions and are prone to overfitting on static regions. This leads to producing results with blurry textures. We consider that decoupling dynamic-static features to enhance dynamic representations can alleviate this issue. Thus, we propose a dynamic-static feature decoupling module (DSFD). Along temporal axes, it regards the regions of current frame features that possess significant differences relative to reference frame features as dynamic features. Conversely, the remaining parts are the static features. Then, we acquire decoupled features driven by dynamic features and current frame features. Moreover, to further enhance the dynamic representation of decoupled features from different viewpoints and ensure accurate motion prediction, we design a temporal-spatial similarity fusion module (TSSF). Along spatial axes, it adaptively selects similar information of dynamic regions. Hinging on the above, we construct a novel approach, DS4D. Experimental results verify our method achieves state-of-the-art (SOTA) results in video-to-4D. In addition, the experiments on a real-world scenario dataset demonstrate its effectiveness on the 4D scene. Our code will be publicly available.