ActionSwitch: Class-agnostic Detection of Simultaneous Actions in Streaming Videos

TL;DR

ActionSwitch is a novel class-agnostic framework for real-time detection of overlapping actions in streaming videos, overcoming class dependency limitations and achieving state-of-the-art results on complex datasets.

Contribution

It introduces the first class-agnostic On-TAL method capable of detecting overlapping actions, with a new conservativeness loss for improved decision-making.

Findings

Achieves state-of-the-art performance on Epic-Kitchens 100.

Effective in detecting overlapping actions of the same class.

Performs well on fine-grained action datasets.

Abstract

Online Temporal Action Localization (On-TAL) is a critical task that aims to instantaneously identify action instances in untrimmed streaming videos as soon as an action concludes -- a major leap from frame-based Online Action Detection (OAD). Yet, the challenge of detecting overlapping actions is often overlooked even though it is a common scenario in streaming videos. Current methods that can address concurrent actions depend heavily on class information, limiting their flexibility. This paper introduces ActionSwitch, the first class-agnostic On-TAL framework capable of detecting overlapping actions. By obviating the reliance on class information, ActionSwitch provides wider applicability to various situations, including overlapping actions of the same class or scenarios where class information is unavailable. This approach is complemented by the proposed "conservativeness loss", which directly embeds a conservative decision-making principle into the loss function for On-TAL. Our ActionSwitch achieves state-of-the-art performance in complex datasets, including Epic-Kitchens 100 targeting the challenging egocentric view and FineAction consisting of fine-grained actions.