Long-Term 3D Point Tracking By Cost Volume Fusion

TL;DR

This paper introduces a novel deep learning framework for long-term 3D point tracking that effectively fuses past information using a transformer-based cost volume, outperforming previous 2D and scene flow methods.

Contribution

It presents the first 3D deep learning approach for long-term point tracking that generalizes without test-time fine-tuning, utilizing a cost volume fusion module with transformer architecture.

Findings

Outperforms simple scene flow chaining in 3D tracking.

Surpasses previous 2D point tracking methods in accuracy.

Works effectively without test-time fine-tuning.

Abstract

Long-term point tracking is essential to understand non-rigid motion in the physical world better. Deep learning approaches have recently been incorporated into long-term point tracking, but most prior work predominantly functions in 2D. Although these methods benefit from the well-established backbones and matching frameworks, the motions they produce do not always make sense in the 3D physical world. In this paper, we propose the first deep learning framework for long-term point tracking in 3D that generalizes to new points and videos without requiring test-time fine-tuning. Our model contains a cost volume fusion module that effectively integrates multiple past appearances and motion information via a transformer architecture, significantly enhancing overall tracking performance. In terms of 3D tracking performance, our model significantly outperforms simple scene flow chaining and previous 2D point tracking methods, even if one uses ground truth depth and camera pose to backproject 2D point tracks in a synthetic scenario.