Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos

TL;DR

This paper introduces a weakly supervised spatio-temporal action detection model that uses only video-level labels and incorporates uncertainty estimation, achieving state-of-the-art results on benchmark datasets.

Contribution

It presents a novel probabilistic MIL framework with uncertainty estimation for weakly supervised action detection in untrimmed videos.

Findings

First weakly-supervised results on AVA dataset

State-of-the-art weakly-supervised results on UCF101-24

Effective handling of MIL assumption violations

Abstract

Despite the recent advances in video classification, progress in spatio-temporal action recognition has lagged behind. A major contributing factor has been the prohibitive cost of annotating videos frame-by-frame. In this paper, we present a spatio-temporal action recognition model that is trained with only video-level labels, which are significantly easier to annotate. Our method leverages per-frame person detectors which have been trained on large image datasets within a Multiple Instance Learning framework. We show how we can apply our method in cases where the standard Multiple Instance Learning assumption, that each bag contains at least one instance with the specified label, is invalid using a novel probabilistic variant of MIL where we estimate the uncertainty of each prediction. Furthermore, we report the first weakly-supervised results on the AVA dataset and state-of-the-art results among weakly-supervised methods on UCF101-24.