MeMSVD: Long-Range Temporal Structure Capturing Using Incremental SVD

TL;DR

This paper introduces MeMSVD, a low-rank incremental SVD method for long-term video understanding that reduces complexity and memory usage while maintaining or improving accuracy over attention-based approaches.

Contribution

The paper proposes a novel incremental SVD-based scheme for long-range temporal modeling that is more efficient and scalable than traditional attention mechanisms.

Findings

Reduces complexity of long-term memory modeling by over an order of magnitude.

Achieves comparable or better accuracy than attention-based methods.

Outperforms state-of-the-art on three long-term video datasets.

Abstract

This paper is on long-term video understanding where the goal is to recognise human actions over long temporal windows (up to minutes long). In prior work, long temporal context is captured by constructing a long-term memory bank consisting of past and future video features which are then integrated into standard (short-term) video recognition backbones through the use of attention mechanisms. Two well-known problems related to this approach are the quadratic complexity of the attention operation and the fact that the whole feature bank must be stored in memory for inference. To address both issues, we propose an alternative to attention-based schemes which is based on a low-rank approximation of the memory obtained using Singular Value Decomposition. Our scheme has two advantages: (a) it reduces complexity by more than an order of magnitude, and (b) it is amenable to an efficient implementation for the calculation of the memory bases in an incremental fashion which does not require the storage of the whole feature bank in memory. The proposed scheme matches or surpasses the accuracy achieved by attention-based mechanisms while being memory-efficient. Through extensive experiments, we demonstrate that our framework generalises to different architectures and tasks, outperforming the state-of-the-art in three datasets.