Tracking by Predicting 3-D Gaussians Over Time

TL;DR

This paper introduces Video-GMAE, a self-supervised method that encodes videos as moving Gaussian splats, enabling effective zero-shot and fine-tuned video tracking by leveraging 3-D scene representations.

Contribution

It presents a novel self-supervised framework that encodes videos as 3-D Gaussian representations, improving tracking and action recognition performance.

Findings

Zero-shot tracking performance comparable to state-of-the-art.

34.6% improvement on Kinetics after fine-tuning.

13.1% improvement on Kubric after fine-tuning.

Abstract

We propose Video Gaussian Masked Autoencoders (Video-GMAE), a self-supervised approach for representation learning that encodes a sequence of images into a set of Gaussian splats moving over time. Representing a video as a set of Gaussians enforces a reasonable inductive bias: that 2-D videos are often consistent projections of a dynamic 3-D scene. We find that tracking emerges when pretraining a network with this architecture. Mapping the trajectory of the learnt Gaussians onto the image plane gives zero-shot tracking performance comparable to state-of-the-art. With small-scale finetuning, our models achieve 34.6% improvement on Kinetics, and 13.1% on Kubric datasets, surpassing existing self-supervised video approaches. The project page and code are publicly available at https://videogmae.org/ and https://github.com/tekotan/video-gmae.