Exploring Temporally-Aware Features for Point Tracking

TL;DR

This paper introduces Chrono, a temporally-aware feature backbone for point tracking that leverages pre-trained representations and a temporal adapter to achieve state-of-the-art results without refinement, improving efficiency and robustness.

Contribution

The paper presents Chrono, a novel backbone with built-in temporal awareness for point tracking, reducing reliance on multi-stage refinement and enhancing performance with pre-trained features.

Findings

Chrono achieves state-of-the-art results on TAP-Vid-DAVIS and TAP-Vid-Kinetics datasets.

Chrono outperforms common feature backbones in point tracking tasks.

Chrono operates efficiently without the need for a refinement stage.

Abstract

Point tracking in videos is a fundamental task with applications in robotics, video editing, and more. While many vision tasks benefit from pre-trained feature backbones to improve generalizability, point tracking has primarily relied on simpler backbones trained from scratch on synthetic data, which may limit robustness in real-world scenarios. Additionally, point tracking requires temporal awareness to ensure coherence across frames, but using temporally-aware features is still underexplored. Most current methods often employ a two-stage process: an initial coarse prediction followed by a refinement stage to inject temporal information and correct errors from the coarse stage. These approach, however, is computationally expensive and potentially redundant if the feature backbone itself captures sufficient temporal information. In this work, we introduce Chrono, a feature backbone specifically designed for point tracking with built-in temporal awareness. Leveraging pre-trained representations from self-supervised learner DINOv2 and enhanced with a temporal adapter, Chrono effectively captures long-term temporal context, enabling precise prediction even without the refinement stage. Experimental results demonstrate that Chrono achieves state-of-the-art performance in a refiner-free setting on the TAP-Vid-DAVIS and TAP-Vid-Kinetics datasets, among common feature backbones used in point tracking as well as DINOv2, with exceptional efficiency. Project page: https://cvlab-kaist.github.io/Chrono/