SWiT-4D: Sliding-Window Transformer for Lossless and Parameter-Free Temporal 4D Generation

TL;DR

SWiT-4D introduces a lossless, parameter-free transformer framework that effectively converts monocular videos into high-quality 4D meshes, leveraging image-to-3D priors with minimal supervision for robust temporal 4D generation.

Contribution

It presents SWiT-4D, a novel sliding-window transformer that integrates with existing image-to-3D models to enable lossless, parameter-free temporal 4D mesh reconstruction from videos.

Findings

Achieves high-fidelity 4D meshes with only short video fine-tuning.

Outperforms existing methods in temporal smoothness and stability.

Demonstrates strong data efficiency and generalization across benchmarks.

Abstract

Despite significant progress in 4D content generation, the conversion of monocular videos into high-quality animated 3D assets with explicit 4D meshes remains considerably challenging. The scarcity of large-scale, naturally captured 4D mesh datasets further limits the ability to train generalizable video-to-4D models from scratch in a purely data-driven manner. Meanwhile, advances in image-to-3D generation, supported by extensive datasets, offer powerful prior models that can be leveraged. To better utilize these priors while minimizing reliance on 4D supervision, we introduce SWiT-4D, a Sliding-Window Transformer for lossless, parameter-free temporal 4D mesh generation. SWiT-4D integrates seamlessly with any Diffusion Transformer (DiT)-based image-to-3D generator, adding spatial-temporal modeling across video frames while preserving the original single-image forward process, enabling 4D mesh reconstruction from videos of arbitrary length. To recover global translation, we further introduce an optimization-based trajectory module tailored for static-camera monocular videos. SWiT-4D demonstrates strong data efficiency: with only a single short (<10s) video for fine-tuning, it achieves high-fidelity geometry and stable temporal consistency, indicating practical deployability under extremely limited 4D supervision. Comprehensive experiments on both in-domain zoo-test sets and challenging out-of-domain benchmarks (C4D, Objaverse, and in-the-wild videos) show that SWiT-4D consistently outperforms existing baselines in temporal smoothness. Project page: https://animotionlab.github.io/SWIT4D/