# Spatio-Temporal Action Localization in a Weakly Supervised Setting

## TL;DR

This paper introduces a weakly supervised method for spatio-temporal action localization in videos, using unsupervised segmentation, CNN feature extraction, and a novel MIL formulation with continuous labels to reduce annotation requirements.

## Contribution

It proposes a new MIL approach with continuous labels and set splitting regularization for weakly supervised action detection in videos.

## Key findings

- Effective on UCF-Sports dataset
- Outperforms some existing weakly supervised methods
- Demonstrates robustness with limited annotations

## Abstract

Enabling computational systems with the ability to localize actions in video-based content has manifold applications. Traditionally, such a problem is approached in a fully-supervised setting where video-clips with complete frame-by-frame annotations around the actions of interest are provided for training. However, the data requirements needed to achieve adequate generalization in this setting is prohibitive. In this work, we circumvent this issue by casting the problem in a weakly supervised setting, i.e., by considering videos as labelled `sets' of unlabelled video segments. Firstly, we apply unsupervised segmentation to take advantage of the elementary structure of each video. Subsequently, a convolutional neural network is used to extract RGB features from the resulting video segments. Finally, Multiple Instance Learning (MIL) is employed to predict labels at the video segment level, thus inherently performing spatio-temporal action detection. In contrast to previous work, we make use of a different MIL formulation in which the label of each video segment is continuous rather then discrete, making the resulting optimization function tractable. Additionally, we utilize a set splitting technique for regularization. Experimental results considering multiple performance indicators on the UCF-Sports data-set support the effectiveness of our approach.