Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptation

TL;DR

This paper introduces Interactive Tracking, a human-in-the-loop paradigm with a new benchmark, evaluation protocol, and a memory-augmented baseline, addressing limitations of existing non-interactive visual trackers.

Contribution

It presents the first large-scale benchmark for interactive tracking, evaluates existing methods, and proposes a novel memory-augmented tracker that learns from user feedback.

Findings

State-of-the-art trackers perform poorly in interactive scenarios.

Strong performance on traditional benchmarks does not transfer to interactive settings.

The proposed IMAT baseline effectively incorporates user feedback for adaptive tracking.

Abstract

Existing visual trackers mainly operate in a non-interactive, fire-and-forget manner, making them impractical for real-world scenarios that require human-in-the-loop adaptation. To overcome this limitation, we introduce Interactive Tracking, a new paradigm that allows users to guide the tracker at any time using natural language commands. To support research in this direction, we make three main contributions. First, we present InteractTrack, the first large-scale benchmark for interactive tracking, containing 150 videos with dense bounding box annotations and timestamped language instructions. Second, we propose a comprehensive evaluation protocol and evaluate 25 representative trackers, showing that state-of-the-art methods fail in interactive scenarios; strong performance on conventional benchmarks does not transfer. Third, we introduce Interactive Memory-Augmented Tracking (IMAT), a new baseline that employs a dynamic memory mechanism to learn from user feedback and update tracking behavior accordingly. Our benchmark, protocol, and baseline establish a foundation for developing more intelligent, adaptive, and collaborative tracking systems, bridging the gap between automated perception and human guidance. The full benchmark, tracking results, and analysis are available at https://github.com/NorahGreen/InteractTrack.git.