Weakly-Supervised Multi-Person Action Recognition in 360$^{\circ}$ Videos

TL;DR

This paper introduces a weakly-supervised framework for recognizing multiple actions in 360-degree videos, transforming videos into panoramic views and using multi-instance multi-label learning, supported by a new dataset.

Contribution

It presents the first omnidirectional video dataset for multi-person action recognition and a novel weakly-supervised method leveraging panoramic transformation and multi-label learning.

Findings

Effective action recognition in 360° videos demonstrated

Model successfully localizes multiple actions

Proves weak supervision suffices for complex scene understanding

Abstract

The recent development of commodity 360$^{\circ}$ cameras have enabled a single video to capture an entire scene, which endows promising potentials in surveillance scenarios. However, research in omnidirectional video analysis has lagged behind the hardware advances. In this work, we address the important problem of action recognition in top-view 360$^{\circ}$ videos. Due to the wide filed-of-view, 360$^{\circ}$ videos usually capture multiple people performing actions at the same time. Furthermore, the appearance of people are deformed. The proposed framework first transforms omnidirectional videos into panoramic videos, then it extracts spatial-temporal features using region-based 3D CNNs for action recognition. We propose a weakly-supervised method based on multi-instance multi-label learning, which trains the model to recognize and localize multiple actions in a video using only video-level action labels as supervision. We perform experiments to quantitatively validate the efficacy of the proposed method and qualitatively demonstrate action localization results. To enable research in this direction, we introduce 360Action, the first omnidirectional video dataset for multi-person action recognition.