HAT: History-Augmented Anchor Transformer for Online Temporal Action Localization

TL;DR

The paper introduces HAT, a transformer-based framework that incorporates historical context to improve online temporal action localization, especially in procedural and egocentric videos, outperforming existing methods.

Contribution

It presents a novel History-Augmented Anchor Transformer that effectively combines long-term and short-term information for better action localization.

Findings

Outperforms state-of-the-art on PREGO datasets

Achieves comparable performance on non-PREGO datasets

Highlights the importance of long-term history in egocentric scenarios

Abstract

Online video understanding often relies on individual frames, leading to frame-by-frame predictions. Recent advancements such as Online Temporal Action Localization (OnTAL), extend this approach to instance-level predictions. However, existing methods mainly focus on short-term context, neglecting historical information. To address this, we introduce the History-Augmented Anchor Transformer (HAT) Framework for OnTAL. By integrating historical context, our framework enhances the synergy between long-term and short-term information, improving the quality of anchor features crucial for classification and localization. We evaluate our model on both procedural egocentric (PREGO) datasets (EGTEA and EPIC) and standard non-PREGO OnTAL datasets (THUMOS and MUSES). Results show that our model outperforms state-of-the-art approaches significantly on PREGO datasets and achieves comparable or slightly superior performance on non-PREGO datasets, underscoring the importance of leveraging long-term history, especially in procedural and egocentric action scenarios. Code is available at: https://github.com/sakibreza/ECCV24-HAT/