AllTracker: Efficient Dense Point Tracking at High Resolution

TL;DR

AllTracker is a novel high-resolution dense point tracking model that estimates long-range correspondences across video frames, combining optical flow and point tracking techniques for superior accuracy and efficiency.

Contribution

We introduce AllTracker, a new architecture that achieves dense, high-resolution point tracking over many frames, trained jointly on optical flow and point tracking datasets.

Findings

State-of-the-art accuracy on high-resolution point tracking

Efficient model with 16 million parameters

Effective joint training on optical flow and point tracking datasets

Abstract

We introduce AllTracker: a model that estimates long-range point tracks by way of estimating the flow field between a query frame and every other frame of a video. Unlike existing point tracking methods, our approach delivers high-resolution and dense (all-pixel) correspondence fields, which can be visualized as flow maps. Unlike existing optical flow methods, our approach corresponds one frame to hundreds of subsequent frames, rather than just the next frame. We develop a new architecture for this task, blending techniques from existing work in optical flow and point tracking: the model performs iterative inference on low-resolution grids of correspondence estimates, propagating information spatially via 2D convolution layers, and propagating information temporally via pixel-aligned attention layers. The model is fast and parameter-efficient (16 million parameters), and delivers state-of-the-art point tracking accuracy at high resolution (i.e., tracking 768x1024 pixels, on a 40G GPU). A benefit of our design is that we can train jointly on optical flow datasets and point tracking datasets, and we find that doing so is crucial for top performance. We provide an extensive ablation study on our architecture details and training recipe, making it clear which details matter most. Our code and model weights are available at https://alltracker.github.io