Harnessing Temporal Causality for Advanced Temporal Action Detection

TL;DR

This paper introduces CausalTAD, a novel approach leveraging temporal causality in videos to improve action detection accuracy by focusing on past or future context, achieving state-of-the-art results.

Contribution

The paper proposes a causality-based method for temporal action detection, emphasizing causal attention to enhance model performance over existing techniques.

Findings

Achieved 1st place in multiple EPIC-Kitchens Challenge 2024 tracks.

Outperformed previous models on several benchmark datasets.

Demonstrated the effectiveness of causal modeling in temporal action detection.

Abstract

As a fundamental task in long-form video understanding, temporal action detection (TAD) aims to capture inherent temporal relations in untrimmed videos and identify candidate actions with precise boundaries. Over the years, various networks, including convolutions, graphs, and transformers, have been explored for effective temporal modeling for TAD. However, these modules typically treat past and future information equally, overlooking the crucial fact that changes in action boundaries are essentially causal events. Inspired by this insight, we propose leveraging the temporal causality of actions to enhance TAD representation by restricting the model's access to only past or future context. We introduce CausalTAD, which combines causal attention and causal Mamba to achieve state-of-the-art performance on multiple benchmarks. Notably, with CausalTAD, we ranked 1st in the Action Recognition, Action Detection, and Audio-Based Interaction Detection tracks at the EPIC-Kitchens Challenge 2024, as well as 1st in the Moment Queries track at the Ego4D Challenge 2024. Our code is available at https://github.com/sming256/OpenTAD/.