Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization

TL;DR

Re2TAL introduces a reversible backbone architecture for video action localization, enabling end-to-end training on long videos with limited memory, and achieves state-of-the-art results on major benchmarks.

Contribution

The paper proposes a novel network rewiring mechanism to convert residual modules into reversible ones, facilitating memory-efficient training and leveraging pre-trained models for improved TAL performance.

Findings

Re2TAL achieves 37.01% mAP on ActivityNet-v1.3.

Re2TAL reaches 64.9% mAP at tIoU=0.5 on THUMOS-14.

Reversible modules reduce memory usage and training effort.

Abstract

Temporal action localization (TAL) requires long-form reasoning to predict actions of various durations and complex content. Given limited GPU memory, training TAL end to end (i.e., from videos to predictions) on long videos is a significant challenge. Most methods can only train on pre-extracted features without optimizing them for the localization problem, consequently limiting localization performance. In this work, to extend the potential in TAL networks, we propose a novel end-to-end method Re2TAL, which rewires pretrained video backbones for reversible TAL. Re2TAL builds a backbone with reversible modules, where the input can be recovered from the output such that the bulky intermediate activations can be cleared from memory during training. Instead of designing one single type of reversible module, we propose a network rewiring mechanism, to transform any module with a residual connection to a reversible module without changing any parameters. This provides two benefits: (1) a large variety of reversible networks are easily obtained from existing and even future model designs, and (2) the reversible models require much less training effort as they reuse the pre-trained parameters of their original non-reversible versions. Re2TAL, only using the RGB modality, reaches 37.01% average mAP on ActivityNet-v1.3, a new state-of-the-art record, and mAP 64.9% at tIoU=0.5 on THUMOS-14, outperforming all other RGB-only methods.