Real-time Online Video Detection with Temporal Smoothing Transformers

TL;DR

This paper introduces TeSTra, a novel transformer-based model for real-time video recognition that efficiently captures long-term video dynamics using temporal smoothing kernels, achieving state-of-the-art results with constant computational overhead.

Contribution

The paper proposes a new temporal smoothing attention mechanism for transformers, enabling constant-time updates and improved long-term video modeling in streaming recognition.

Findings

TeSTra runs 6 times faster than traditional sliding-window transformers.

Achieves state-of-the-art results on THUMOS'14 and EPIC-Kitchen-100 datasets.

Real-time TeSTra outperforms most prior methods on online action detection.

Abstract

Streaming video recognition reasons about objects and their actions in every frame of a video. A good streaming recognition model captures both long-term dynamics and short-term changes of video. Unfortunately, in most existing methods, the computational complexity grows linearly or quadratically with the length of the considered dynamics. This issue is particularly pronounced in transformer-based architectures. To address this issue, we reformulate the cross-attention in a video transformer through the lens of kernel and apply two kinds of temporal smoothing kernel: A box kernel or a Laplace kernel. The resulting streaming attention reuses much of the computation from frame to frame, and only requires a constant time update each frame. Based on this idea, we build TeSTra, a Temporal Smoothing Transformer, that takes in arbitrarily long inputs with constant caching and computing overhead. Specifically, it runs $6\times$ faster than equivalent sliding-window based transformers with 2,048 frames in a streaming setting. Furthermore, thanks to the increased temporal span, TeSTra achieves state-of-the-art results on THUMOS'14 and EPIC-Kitchen-100, two standard online action detection and action anticipation datasets. A real-time version of TeSTra outperforms all but one prior approaches on the THUMOS'14 dataset.