Approximated Bilinear Modules for Temporal Modeling

TL;DR

This paper introduces approximated bilinear modules (ABMs) for efficient and effective temporal modeling in videos, capturing fine-grained cues and reasoning, and demonstrates superior performance on action recognition datasets.

Contribution

It proposes a novel ABM approach that leverages two-layer MLPs as bilinear approximations, enabling deep temporal modeling within CNNs while reusing pretrained parameters.

Findings

Outperforms state-of-the-art on Something-Something v1 and v2 datasets.

Achieves high efficiency through snippet sampling and shifting inference.

Demonstrates effectiveness of ABMs via extensive ablation studies.

Abstract

We consider two less-emphasized temporal properties of video: 1. Temporal cues are fine-grained; 2. Temporal modeling needs reasoning. To tackle both problems at once, we exploit approximated bilinear modules (ABMs) for temporal modeling. There are two main points making the modules effective: two-layer MLPs can be seen as a constraint approximation of bilinear operations, thus can be used to construct deep ABMs in existing CNNs while reusing pretrained parameters; frame features can be divided into static and dynamic parts because of visual repetition in adjacent frames, which enables temporal modeling to be more efficient. Multiple ABM variants and implementations are investigated, from high performance to high efficiency. Specifically, we show how two-layer subnets in CNNs can be converted to temporal bilinear modules by adding an auxiliary-branch. Besides, we introduce snippet sampling and shifting inference to boost sparse-frame video classification performance. Extensive ablation studies are conducted to show the effectiveness of proposed techniques. Our models can outperform most state-of-the-art methods on Something-Something v1 and v2 datasets without Kinetics pretraining, and are also competitive on other YouTube-like action recognition datasets. Our code is available on https://github.com/zhuxinqimac/abm-pytorch.