AM Flow: Adapters for Temporal Processing in Action Recognition

TL;DR

This paper introduces AM Flow, a method for identifying motion-relevant pixels in video frames, combined with adapters for efficient temporal processing in image models, leading to faster training and state-of-the-art action recognition results.

Contribution

The paper proposes AM Flow for motion segmentation in videos and extends adapters for efficient temporal processing in pretrained image models, reducing training time and improving accuracy.

Findings

Achieves state-of-the-art results on Kinetics-400 and other datasets.

Reduces training epochs needed for action recognition.

Provides a faster convergence method for video classification.

Abstract

Deep learning models, in particular \textit{image} models, have recently gained generalisability and robustness. %are becoming more general and robust by the day. In this work, we propose to exploit such advances in the realm of \textit{video} classification. Video foundation models suffer from the requirement of extensive pretraining and a large training time. Towards mitigating such limitations, we propose "\textit{Attention Map (AM) Flow}" for image models, a method for identifying pixels relevant to motion in each input video frame. In this context, we propose two methods to compute AM flow, depending on camera motion. AM flow allows the separation of spatial and temporal processing, while providing improved results over combined spatio-temporal processing (as in video models). Adapters, one of the popular techniques in parameter efficient transfer learning, facilitate the incorporation of AM flow into pretrained image models, mitigating the need for full-finetuning. We extend adapters to "\textit{temporal processing adapters}" by incorporating a temporal processing unit into the adapters. Our work achieves faster convergence, therefore reducing the number of epochs needed for training. Moreover, we endow an image model with the ability to achieve state-of-the-art results on popular action recognition datasets. This reduces training time and simplifies pretraining. We present experiments on Kinetics-400, Something-Something v2, and Toyota Smarthome datasets, showcasing state-of-the-art or comparable results.