GOT-Edit: Geometry-Aware Generic Object Tracking via Online Model Editing

TL;DR

GOT-Edit introduces a geometry-aware online model editing method that enhances generic object tracking by integrating 3D geometric cues with 2D semantics, improving robustness under occlusion and clutter.

Contribution

It presents a novel online cross-modality model editing approach that incorporates 3D geometric information into 2D video tracking using a pre-trained Visual Geometry Grounded Transformer.

Findings

Achieves superior robustness and accuracy on multiple GOT benchmarks.

Performs well under occlusion and clutter scenarios.

Demonstrates the effectiveness of combining 2D semantics with 3D geometric reasoning.

Abstract

Human perception for effective object tracking in 2D video streams arises from the implicit use of prior 3D knowledge and semantic reasoning. In contrast, most generic object tracking (GOT) methods primarily rely on 2D features of the target and its surroundings, while neglecting 3D geometric cues, making them susceptible to partial occlusion, distractors, and variations in geometry and appearance. To address this limitation, we introduce GOT-Edit, an online cross-modality model editing approach that integrates geometry-aware cues into a generic object tracker from a 2D video stream. Our approach leverages features from a pre-trained Visual Geometry Grounded Transformer to infer geometric cues from only a few 2D images. To address the challenge of seamlessly combining geometry and semantics, GOT-Edit performs online model editing. By leveraging null-space constraints during model updates, it incorporates geometric information while preserving semantic discrimination, yielding consistently better performance across diverse scenarios. Extensive experiments on multiple GOT benchmarks demonstrate that GOT-Edit achieves superior robustness and accuracy, particularly under occlusion and clutter, establishing a new paradigm for combining 2D semantics with 3D geometric reasoning for generic object tracking. The project page is available at https://chenshihfang.github.io/GOT-EDIT.