Long-term Pre-training for Temporal Action Detection with Transformers

TL;DR

This paper introduces Long-Term Pre-training (LTP), a novel pre-training strategy for transformers in temporal action detection, which synthesizes long-form video features and employs long-term pretext tasks to improve performance and mitigate data scarcity issues.

Contribution

The paper proposes LTP, a new pre-training method with class-wise synthesis and long-term pretext tasks, specifically designed to enhance transformer-based TAD models under data scarcity.

Findings

Achieves state-of-the-art results on ActivityNet-v1.3 and THUMOS14.

Significantly alleviates data scarcity issues in TAD.

Demonstrates the effectiveness of long-term pre-training in transformer models.

Abstract

Temporal action detection (TAD) is challenging, yet fundamental for real-world video applications. Recently, DETR-based models for TAD have been prevailing thanks to their unique benefits. However, transformers demand a huge dataset, and unfortunately data scarcity in TAD causes a severe degeneration. In this paper, we identify two crucial problems from data scarcity: attention collapse and imbalanced performance. To this end, we propose a new pre-training strategy, Long-Term Pre-training (LTP), tailored for transformers. LTP has two main components: 1) class-wise synthesis, 2) long-term pretext tasks. Firstly, we synthesize long-form video features by merging video snippets of a target class and non-target classes. They are analogous to untrimmed data used in TAD, despite being created from trimmed data. In addition, we devise two types of long-term pretext tasks to learn long-term dependency. They impose long-term conditions such as finding second-to-fourth or short-duration actions. Our extensive experiments show state-of-the-art performances in DETR-based methods on ActivityNet-v1.3 and THUMOS14 by a large margin. Moreover, we demonstrate that LTP significantly relieves the data scarcity issues in TAD.