Learning Policies for Adaptive Tracking with Deep Feature Cascades

TL;DR

This paper introduces an adaptive deep tracking method that dynamically chooses between cheap and expensive features for each frame, significantly boosting speed while maintaining high accuracy.

Contribution

It formulates adaptive tracking as a decision-making process and trains an agent via reinforcement learning to select features, enabling near real-time performance on CPU.

Findings

Achieves 23 fps on CPU with state-of-the-art accuracy.

Provides a 100x speedup for nearly half of the frames.

Demonstrates the effectiveness of adaptive feature selection in tracking.

Abstract

Visual object tracking is a fundamental and time-critical vision task. Recent years have seen many shallow tracking methods based on real-time pixel-based correlation filters, as well as deep methods that have top performance but need a high-end GPU. In this paper, we learn to improve the speed of deep trackers without losing accuracy. Our fundamental insight is to take an adaptive approach, where easy frames are processed with cheap features (such as pixel values), while challenging frames are processed with invariant but expensive deep features. We formulate the adaptive tracking problem as a decision-making process, and learn an agent to decide whether to locate objects with high confidence on an early layer, or continue processing subsequent layers of a network. This significantly reduces the feed-forward cost for easy frames with distinct or slow-moving objects. We train the agent offline in a reinforcement learning fashion, and further demonstrate that learning all deep layers (so as to provide good features for adaptive tracking) can lead to near real-time average tracking speed of 23 fps on a single CPU while achieving state-of-the-art performance. Perhaps most tellingly, our approach provides a 100X speedup for almost 50% of the time, indicating the power of an adaptive approach.