4D-Animal: Freely Reconstructing Animatable 3D Animals from Videos

TL;DR

4D-Animal introduces a keypoint-free framework for reconstructing animatable 3D animals from videos, utilizing dense features and hierarchical alignment to improve efficiency, stability, and temporal coherence.

Contribution

It presents a novel keypoint-free approach with dense feature mapping and multi-level alignment strategies for 3D animal reconstruction from videos.

Findings

Outperforms existing model-based and model-free methods.

Produces high-quality, temporally coherent 3D animal reconstructions.

Enables large-scale applications with improved stability and efficiency.

Abstract

Existing methods for reconstructing animatable 3D animals from videos typically rely on sparse semantic keypoints to fit parametric models. However, obtaining such keypoints is labor-intensive, and keypoint detectors trained on limited animal data are often unreliable. To address this, we propose 4D-Animal, a novel framework that reconstructs animatable 3D animals from videos without requiring sparse keypoint annotations. Our approach introduces a dense feature network that maps 2D representations to SMAL parameters, enhancing both the efficiency and stability of the fitting process. Furthermore, we develop a hierarchical alignment strategy that integrates silhouette, part-level, pixel-level, and temporal cues from pre-trained 2D visual models to produce accurate and temporally coherent reconstructions across frames. Extensive experiments demonstrate that 4D-Animal outperforms both model-based and model-free baselines. Moreover, the high-quality 3D assets generated by our method can benefit other 3D tasks, underscoring its potential for large-scale applications. The code is released at https://github.com/zhongshsh/4D-Animal.