Introducing Gating and Context into Temporal Action Detection

TL;DR

This paper enhances temporal action detection in videos by integrating gating and context mechanisms into transformer-based models, leading to improved localization and classification accuracy on challenging datasets.

Contribution

It introduces a novel feature extraction approach with gating and context branches that better capture temporal dependencies and contextual information in untrimmed videos.

Findings

Improved detection accuracy on THUMOS14 dataset.

Enhanced contextual understanding on EPIC-KITCHEN 100.

Consistent performance gains over baseline methods.

Abstract

Temporal Action Detection (TAD), the task of localizing and classifying actions in untrimmed video, remains challenging due to action overlaps and variable action durations. Recent findings suggest that TAD performance is dependent on the structural design of transformers rather than on the self-attention mechanism. Building on this insight, we propose a refined feature extraction process through lightweight, yet effective operations. First, we employ a local branch that employs parallel convolutions with varying window sizes to capture both fine-grained and coarse-grained temporal features. This branch incorporates a gating mechanism to select the most relevant features. Second, we introduce a context branch that uses boundary frames as key-value pairs to analyze their relationship with the central frame through cross-attention. The proposed method captures temporal dependencies and improves contextual understanding. Evaluations of the gating mechanism and context branch on challenging datasets (THUMOS14 and EPIC-KITCHEN 100) show a consistent improvement over the baseline and existing methods.