Long-Term Visual Object Tracking with Event Cameras: An Associative Memory Augmented Tracker and A Benchmark Dataset

TL;DR

This paper introduces a new long-term visual object tracking dataset called FELT, and proposes AMTTrack, a novel RGB-Event tracker utilizing associative memory and transformer architecture, validated through extensive experiments.

Contribution

The paper presents the FELT dataset for long-term tracking and a new associative memory transformer tracker, AMTTrack, addressing appearance variation in long-term scenarios.

Findings

AMTTrack outperforms baseline trackers on multiple datasets.

FELT dataset enables comprehensive evaluation of long-term tracking.

Source code and dataset will be publicly available.

Abstract

Existing event stream based trackers undergo evaluation on short-term tracking datasets, however, the tracking of real-world scenarios involves long-term tracking, and the performance of existing tracking algorithms in these scenarios remains unclear. In this paper, we first propose a new long-term, large-scale frame-event visual object tracking dataset, termed FELT. It contains 1,044 long-term videos that involve 1.9 million RGB frames and event stream pairs, 60 different target objects, and 14 challenging attributes. To build a solid benchmark, we retrain and evaluate 21 baseline trackers on our dataset for future work to compare. In addition, we propose a novel Associative Memory Transformer based RGB-Event long-term visual tracker, termed AMTTrack. It follows a one-stream tracking framework and aggregates the multi-scale RGB/event template and search tokens effectively via the Hopfield retrieval layer. The framework also embodies another aspect of associative memory by maintaining dynamic template representations through an associative memory update scheme, which addresses the appearance variation in long-term tracking. Extensive experiments on FELT, FE108, VisEvent, and COESOT datasets fully validated the effectiveness of our proposed tracker. Both the dataset and source code will be released on https://github.com/Event-AHU/FELT_SOT_Benchmark