Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memories

TL;DR

This paper proposes a collaborative memory mechanism for video models that encodes information across multiple clips, enabling better long-term temporal understanding and improving classification accuracy without significant computational costs.

Contribution

It introduces an end-to-end trainable collaborative memory framework that captures long-range dependencies across clips, outperforming existing methods on multiple video recognition benchmarks.

Findings

Significant accuracy improvements on Kinetics, Charades, and Something-Something datasets.

Effective modeling of long-term temporal dependencies.

Framework generalizes across different video architectures and tasks.

Abstract

The standard way of training video models entails sampling at each iteration a single clip from a video and optimizing the clip prediction with respect to the video-level label. We argue that a single clip may not have enough temporal coverage to exhibit the label to recognize, since video datasets are often weakly labeled with categorical information but without dense temporal annotations. Furthermore, optimizing the model over brief clips impedes its ability to learn long-term temporal dependencies. To overcome these limitations, we introduce a collaborative memory mechanism that encodes information across multiple sampled clips of a video at each training iteration. This enables the learning of long-range dependencies beyond a single clip. We explore different design choices for the collaborative memory to ease the optimization difficulties. Our proposed framework is end-to-end trainable and significantly improves the accuracy of video classification at a negligible computational overhead. Through extensive experiments, we demonstrate that our framework generalizes to different video architectures and tasks, outperforming the state of the art on both action recognition (e.g., Kinetics-400 & 700, Charades, Something-Something-V1) and action detection (e.g., AVA v2.1 & v2.2).