Spatial-Temporal State Propagation Autoregressive Model for 4D Object Generation

TL;DR

This paper introduces 4DSTAR, a novel autoregressive model that generates 4D objects with high spatial-temporal consistency by propagating states across time and space, outperforming diffusion-based methods.

Contribution

The paper proposes a new autoregressive framework with dynamic state propagation and a 4D VQ-VAE for coherent 4D object generation, addressing limitations of existing diffusion models.

Findings

Achieves spatial-temporal consistency in 4D object generation.

Demonstrates performance competitive with diffusion models.

Introduces a dynamic state propagation mechanism for long-term dependencies.

Abstract

Generating high-quality 4D objects with spatial-temporal consistency is still formidable. Existing diffusion-based methods often struggle with spatial-temporal inconsistency, as they fail to leverage outputs from all previous timesteps to guide the generation at the current timestep. Therefore, we propose a Spatial-Temporal State Propagation AutoRegressive Model (4DSTAR), which generates 4D objects maintaining temporal-spatial consistency. 4DSTAR formulates the generation problem as the prediction of tokens that represent the 4D object. It consists of two key components: (1) The dynamic spatial-temporal state propagation autoregressive model (STAR) is proposed, which achieves spatial-temporal consistent generation. Unlike standard autoregressive models, STAR divides prediction tokens into groups based on timesteps. It models long-term dependencies by propagating spatial-temporal states from previous groups and utilizes these dependencies to guide generation at the next timestep. To this end, a spatial-temporal container is proposed, which dynamically updating the effective spatial-temporal state features from all historical groups, then updated features serve as conditional features to guide the prediction of the next token group. (2) The 4D VQ-VAE is proposed, which implicitly encodes the 4D structure into discrete space and decodes the discrete tokens predicted by STAR into temporally coherent dynamic 3D Gaussians. Experiments demonstrate that 4DSTAR generates spatial-temporal consistent 4D objects, and achieves performance competitive with diffusion models.