Actor-centered Representations for Action Localization in Streaming Videos

TL;DR

This paper introduces a hierarchical predictive learning framework for actor-centered action localization in streaming videos, achieving strong results with minimal training and extending to multi-actor scenarios without labels.

Contribution

It presents a novel actor-centered, unsupervised learning approach for action localization that requires only one epoch of training and generalizes well to new domains.

Findings

Outperforms unsupervised and weakly supervised baselines

Achieves competitive results with fully supervised methods

Effectively extends to multi-actor and out-of-domain scenarios

Abstract

Event perception tasks such as recognizing and localizing actions in streaming videos are essential for scaling to real-world application contexts. We tackle the problem of learning actor-centered representations through the notion of continual hierarchical predictive learning to localize actions in streaming videos without the need for training labels and outlines for the objects in the video. We propose a framework driven by the notion of hierarchical predictive learning to construct actor-centered features by attention-based contextualization. The key idea is that predictable features or objects do not attract attention and hence do not contribute to the action of interest. Experiments on three benchmark datasets show that the approach can learn robust representations for localizing actions using only one epoch of training, i.e., a single pass through the streaming video. We show that the proposed approach outperforms unsupervised and weakly supervised baselines while offering competitive performance to fully supervised approaches. Additionally, we extend the model to multi-actor settings to recognize group activities while localizing the multiple, plausible actors. We also show that it generalizes to out-of-domain data with limited performance degradation.