Out of Sight, Out of Track: Adversarial Attacks on Propagation-based Multi-Object Trackers via Query State Manipulation

TL;DR

This paper introduces FADE, an attack framework exploiting vulnerabilities in propagation-based multi-object trackers, demonstrating effective physical-world attacks that cause track disruptions and identity switches.

Contribution

The paper presents FADE, a novel attack method targeting core TBP mechanisms, with a differentiable pipeline for physical-world attack optimization.

Findings

FADE significantly disrupts state-of-the-art TBP trackers.

It causes high identity switch rates and track terminations.

Effective in simulated physical-world scenarios.

Abstract

Recent Tracking-by-Query-Propagation (TBP) methods have advanced Multi-Object Tracking (MOT) by enabling end-to-end (E2E) pipelines with long-range temporal modeling. However, this reliance on query propagation introduces unexplored architectural vulnerabilities to adversarial attacks. We present FADE, a novel attack framework designed to exploit these specific vulnerabilities. FADE employs two attack strategies targeting core TBP mechanisms: (i) Temporal Query Flooding: Generates spurious temporally consistent track queries to exhaust the tracker's limited query budget, forcing it to terminate valid tracks. (ii) Temporal Memory Corruption: Directly attacks the query updater's memory by severing temporal links via state de-correlation and erasing the learned feature identity of matched tracks. Furthermore, we introduce a differentiable pipeline to optimize these attacks for physical-world realizability by leveraging simulations of advanced perception sensor spoofing. Experiments on MOT17 and MOT20 benchmarks demonstrate that FADE is highly effective against state-of-the-art TBP trackers, causing significant identity switches and track terminations.