Multi-label Class-imbalanced Action Recognition in Hockey Videos via 3D Convolutional Neural Networks

TL;DR

This paper presents a 3D CNN-based multi-label deep learning system for recognizing multiple, imbalanced actions in hockey videos, addressing challenges like high variation, class imbalance, and simultaneous actions.

Contribution

It introduces a novel multi-label 3D CNN approach for class-imbalanced action recognition in sports videos, outperforming existing solutions.

Findings

Proposed method outperforms previous system on the dataset.

Ensemble of binary networks vs. single multi-output network comparison.

Effective handling of multi-label and class imbalance challenges.

Abstract

Automatic analysis of the video is one of most complex problems in the fields of computer vision and machine learning. A significant part of this research deals with (human) activity recognition (HAR) since humans, and the activities that they perform, generate most of the video semantics. Video-based HAR has applications in various domains, but one of the most important and challenging is HAR in sports videos. Some of the major issues include high inter- and intra-class variations, large class imbalance, the presence of both group actions and single player actions, and recognizing simultaneous actions, i.e., the multi-label learning problem. Keeping in mind these challenges and the recent success of CNNs in solving various computer vision problems, in this work, we implement a 3D CNN based multi-label deep HAR system for multi-label class-imbalanced action recognition in hockey videos. We test our system for two different scenarios: an ensemble of $k$ binary networks vs. a single $k$-output network, on a publicly available dataset. We also compare our results with the system that was originally designed for the chosen dataset. Experimental results show that the proposed approach performs better than the existing solution.