SimOn: A Simple Framework for Online Temporal Action Localization

TL;DR

SimOn introduces a simple Transformer-based framework for online temporal action localization that effectively predicts action instances from streaming videos without future frame access, outperforming previous methods on benchmark datasets.

Contribution

The paper presents a novel end-to-end Transformer framework for On-TAL that leverages past visual context and learnable embeddings, setting new state-of-the-art results.

Findings

Outperforms previous methods on THUMOS14 and ActivityNet1.3 datasets.

Achieves new state-of-the-art performance in online temporal action localization.

Demonstrates robustness and effectiveness in online detection of action start.

Abstract

Online Temporal Action Localization (On-TAL) aims to immediately provide action instances from untrimmed streaming videos. The model is not allowed to utilize future frames and any processing techniques to modify past predictions, making On-TAL much more challenging. In this paper, we propose a simple yet effective framework, termed SimOn, that learns to predict action instances using the popular Transformer architecture in an end-to-end manner. Specifically, the model takes the current frame feature as a query and a set of past context information as keys and values of the Transformer. Different from the prior work that uses a set of outputs of the model as past contexts, we leverage the past visual context and the learnable context embedding for the current query. Experimental results on the THUMOS14 and ActivityNet1.3 datasets show that our model remarkably outperforms the previous methods, achieving a new state-of-the-art On-TAL performance. In addition, the evaluation for Online Detection of Action Start (ODAS) demonstrates the effectiveness and robustness of our method in the online setting. The code is available at https://github.com/TuanTNG/SimOn