What Can Simple Arithmetic Operations Do for Temporal Modeling?

TL;DR

This paper introduces a simple, plug-and-play Arithmetic Temporal Module (ATM) that uses basic arithmetic operations on frame features to enhance temporal modeling in videos, achieving high accuracy with low computational cost.

Contribution

The work proposes a novel ATM that leverages simple arithmetic operations for effective temporal modeling, compatible with CNNs and ViTs, and demonstrates superior performance on video benchmarks.

Findings

ATM improves temporal modeling with low computational cost.

Achieves state-of-the-art accuracy on video benchmarks.

Compatible with various neural network architectures.

Abstract

Temporal modeling plays a crucial role in understanding video content. To tackle this problem, previous studies built complicated temporal relations through time sequence thanks to the development of computationally powerful devices. In this work, we explore the potential of four simple arithmetic operations for temporal modeling. Specifically, we first capture auxiliary temporal cues by computing addition, subtraction, multiplication, and division between pairs of extracted frame features. Then, we extract corresponding features from these cues to benefit the original temporal-irrespective domain. We term such a simple pipeline as an Arithmetic Temporal Module (ATM), which operates on the stem of a visual backbone with a plug-and-play style. We conduct comprehensive ablation studies on the instantiation of ATMs and demonstrate that this module provides powerful temporal modeling capability at a low computational cost. Moreover, the ATM is compatible with both CNNs- and ViTs-based architectures. Our results show that ATM achieves superior performance over several popular video benchmarks. Specifically, on Something-Something V1, V2 and Kinetics-400, we reach top-1 accuracy of 65.6%, 74.6%, and 89.4% respectively. The code is available at https://github.com/whwu95/ATM.