Learning Dynamic Memory Networks for Object Tracking

TL;DR

This paper introduces a dynamic memory network for object tracking that adapts to appearance changes in real-time using an LSTM-based memory controller, attention mechanisms, and gated residual learning, achieving high accuracy and speed.

Contribution

It proposes a novel feed-forward tracking approach with external memory, enabling efficient adaptation without online fine-tuning and scalable memory capacity.

Findings

Outperforms state-of-the-art on OTB and VOT datasets

Operates at 50 fps in real-time

Effectively adapts to appearance variations

Abstract

Template-matching methods for visual tracking have gained popularity recently due to their comparable performance and fast speed. However, they lack effective ways to adapt to changes in the target object's appearance, making their tracking accuracy still far from state-of-the-art. In this paper, we propose a dynamic memory network to adapt the template to the target's appearance variations during tracking. An LSTM is used as a memory controller, where the input is the search feature map and the outputs are the control signals for the reading and writing process of the memory block. As the location of the target is at first unknown in the search feature map, an attention mechanism is applied to concentrate the LSTM input on the potential target. To prevent aggressive model adaptivity, we apply gated residual template learning to control the amount of retrieved memory that is used to combine with the initial template. Unlike tracking-by-detection methods where the object's information is maintained by the weight parameters of neural networks, which requires expensive online fine-tuning to be adaptable, our tracker runs completely feed-forward and adapts to the target's appearance changes by updating the external memory. Moreover, unlike other tracking methods where the model capacity is fixed after offline training --- the capacity of our tracker can be easily enlarged as the memory requirements of a task increase, which is favorable for memorizing long-term object information. Extensive experiments on OTB and VOT demonstrates that our tracker MemTrack performs favorably against state-of-the-art tracking methods while retaining real-time speed of 50 fps.