Fine-grained Action Segmentation using the Semi-Supervised Action GAN

TL;DR

This paper introduces a semi-supervised GAN model with a novel Gated Context Extractor for fine-grained action segmentation in videos, effectively capturing hierarchical actions and transitions.

Contribution

It presents a new recurrent semi-supervised GAN architecture with a Gated Context Extractor for improved action segmentation in unsegmented videos.

Findings

Outperforms state-of-the-art on three challenging datasets

Effectively captures hierarchical action transitions

Demonstrates the importance of the Gated Context Extractor

Abstract

In this paper we address the problem of continuous fine-grained action segmentation, in which multiple actions are present in an unsegmented video stream. The challenge for this task lies in the need to represent the hierarchical nature of the actions and to detect the transitions between actions, allowing us to localise the actions within the video effectively. We propose a novel recurrent semi-supervised Generative Adversarial Network (GAN) model for continuous fine-grained human action segmentation. Temporal context information is captured via a novel Gated Context Extractor (GCE) module, composed of gated attention units, that directs the queued context information through the generator model, for enhanced action segmentation. The GAN is made to learn features in a semi-supervised manner, enabling the model to perform action classification jointly with the standard, unsupervised, GAN learning procedure. We perform extensive evaluations on different architectural variants to demonstrate the importance of the proposed network architecture, and show that it is capable of outperforming current state-of-the-art on three challenging datasets: 50 Salads, MERL Shopping and Georgia Tech Egocentric Activities dataset.