Velox: Learning Representations of 4D Geometry and Appearance

TL;DR

Velox introduces a novel framework for learning compact, descriptive, and accessible 4D object representations from minimal input, enabling efficient downstream tasks like video generation and 3D tracking.

Contribution

The paper presents Velox, a method that encodes 4D objects into dynamic shape tokens using dual decoders, improving efficiency and versatility in representing 4D geometry and appearance.

Findings

Strong performance in video-to-4D generation.

Effective 3D tracking and cloth simulation.

Compact representations facilitate downstream tasks.

Abstract

We introduce a framework for learning latent representations of 4D objects which are descriptive, faithfully capturing object geometry and appearance; compressive, aiding in downstream efficiency; and accessible, requiring minimal input, i.e., an unstructured dynamic point cloud, to construct. Specifically, Velox trains an encoder to compress spatiotemporal color point clouds into a set of dynamic shape tokens. These tokens are supervised using two complementary decoders: a 4D surface decoder, which models the time-varying surface distribution capturing the geometry; and a Gaussian decoder, which maps the tokens to 3D Gaussians, helping learn appearance. To demonstrate the utility of our representation, we evaluate it across three downstream tasks -- video-to-4D generation, 3D tracking, and cloth simulation via image-to-4D generation -- and observe strong performances in all settings.