Covariance of Motion and Appearance Featuresfor Spatio Temporal Recognition Tasks

TL;DR

This paper presents a novel covariance-based video analysis framework that combines motion and appearance features for improved spatio-temporal recognition, applicable to event and gesture recognition tasks.

Contribution

The paper introduces a new covariance descriptor for video analysis, integrating motion and appearance features, and formulates sparse recognition as a determinant maximization problem on Riemannian manifolds.

Findings

Effective in low-level event recognition in unconstrained scenarios

Successful in gesture recognition with one-shot learning

Provides promising results for large-scale video analysis

Abstract

In this paper, we introduce an end-to-end framework for video analysis focused towards practical scenarios built on theoretical foundations from sparse representation, including a novel descriptor for general purpose video analysis. In our approach, we compute kinematic features from optical flow and first and second-order derivatives of intensities to represent motion and appearance respectively. These features are then used to construct covariance matrices which capture joint statistics of both low-level motion and appearance features extracted from a video. Using an over-complete dictionary of the covariance based descriptors built from labeled training samples, we formulate low-level event recognition as a sparse linear approximation problem. Within this, we pose the sparse decomposition of a covariance matrix, which also conforms to the space of semi-positive definite matrices, as a determinant maximization problem. Also since covariance matrices lie on non-linear Riemannian manifolds, we compare our former approach with a sparse linear approximation alternative that is suitable for equivalent vector spaces of covariance matrices. This is done by searching for the best projection of the query data on a dictionary using an Orthogonal Matching pursuit algorithm. We show the applicability of our video descriptor in two different application domains - namely low-level event recognition in unconstrained scenarios and gesture recognition using one shot learning. Our experiments provide promising insights in large scale video analysis.