An end-to-end generative framework for video segmentation and recognition

TL;DR

This paper presents an end-to-end generative framework combining Fisher Vector-based visual representations with structured temporal models for improved video activity segmentation and recognition across diverse datasets.

Contribution

It introduces a novel generative approach that effectively integrates Fisher Vectors with structured temporal models for activity recognition and parsing.

Findings

Outperforms state-of-the-art methods on large datasets

Effective for complex activities and action unit parsing

Utilizes Fisher Vectors as a suitable front-end for generative models

Abstract

We describe an end-to-end generative approach for the segmentation and recognition of human activities. In this approach, a visual representation based on reduced Fisher Vectors is combined with a structured temporal model for recognition. We show that the statistical properties of Fisher Vectors make them an especially suitable front-end for generative models such as Gaussian mixtures. The system is evaluated for both the recognition of complex activities as well as their parsing into action units. Using a variety of video datasets ranging from human cooking activities to animal behaviors, our experiments demonstrate that the resulting architecture outperforms state-of-the-art approaches for larger datasets, i.e. when sufficient amount of data is available for training structured generative models.