SHaDe: Compact and Consistent Dynamic 3D Reconstruction via Tri-Plane Deformation and Latent Diffusion

TL;DR

SHaDe introduces a compact, efficient framework for dynamic 3D scene reconstruction using tri-plane deformation, view-conditioned radiance fields, and latent diffusion, achieving state-of-the-art results in quality and consistency.

Contribution

The paper proposes a novel combination of tri-plane encoding, explicit deformation, SH-based rendering, and latent diffusion for improved dynamic 3D reconstruction.

Findings

Outperforms recent methods like HexPlane and 4D Gaussian Splatting in visual quality.

Achieves superior temporal coherence and robustness to sparse views.

Provides a more interpretable and efficient reconstruction pipeline.

Abstract

We present a novel framework for dynamic 3D scene reconstruction that integrates three key components: an explicit tri-plane deformation field, a view-conditioned canonical radiance field with spherical harmonics (SH) attention, and a temporally-aware latent diffusion prior. Our method encodes 4D scenes using three orthogonal 2D feature planes that evolve over time, enabling efficient and compact spatiotemporal representation. These features are explicitly warped into a canonical space via a deformation offset field, eliminating the need for MLP-based motion modeling. In canonical space, we replace traditional MLP decoders with a structured SH-based rendering head that synthesizes view-dependent color via attention over learned frequency bands improving both interpretability and rendering efficiency. To further enhance fidelity and temporal consistency, we introduce a transformer-guided latent diffusion module that refines the tri-plane and deformation features in a compressed latent space. This generative module denoises scene representations under ambiguous or out-of-distribution (OOD) motion, improving generalization. Our model is trained in two stages: the diffusion module is first pre-trained independently, and then fine-tuned jointly with the full pipeline using a combination of image reconstruction, diffusion denoising, and temporal consistency losses. We demonstrate state-of-the-art results on synthetic benchmarks, surpassing recent methods such as HexPlane and 4D Gaussian Splatting in visual quality, temporal coherence, and robustness to sparse-view dynamic inputs.