AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognition

TL;DR

AdaFocus V2 introduces an end-to-end trainable, efficient spatial dynamic network for video recognition, improving accuracy and training simplicity over the original AdaFocus by reformulating its training process.

Contribution

It presents a differentiable, one-stage training method for AdaFocus, along with an improved training scheme and a conditional-exit technique for adaptive computation.

Findings

Outperforms original AdaFocus and baselines on six datasets.

Significantly more efficient and easier to train.

Achieves better accuracy with reduced training complexity.

Abstract

Recent works have shown that the computational efficiency of video recognition can be significantly improved by reducing the spatial redundancy. As a representative work, the adaptive focus method (AdaFocus) has achieved a favorable trade-off between accuracy and inference speed by dynamically identifying and attending to the informative regions in each video frame. However, AdaFocus requires a complicated three-stage training pipeline (involving reinforcement learning), leading to slow convergence and is unfriendly to practitioners. This work reformulates the training of AdaFocus as a simple one-stage algorithm by introducing a differentiable interpolation-based patch selection operation, enabling efficient end-to-end optimization. We further present an improved training scheme to address the issues introduced by the one-stage formulation, including the lack of supervision, input diversity and training stability. Moreover, a conditional-exit technique is proposed to perform temporal adaptive computation on top of AdaFocus without additional training. Extensive experiments on six benchmark datasets (i.e., ActivityNet, FCVID, Mini-Kinetics, Something-Something V1&V2, and Jester) demonstrate that our model significantly outperforms the original AdaFocus and other competitive baselines, while being considerably more simple and efficient to train. Code is available at https://github.com/LeapLabTHU/AdaFocusV2.