Grassmannian learning mutual subspace method for image set recognition

TL;DR

This paper introduces G-LMSM, a novel neural network layer that improves image set recognition by representing sets as subspaces on the Grassmann manifold and matching them via canonical angles, trained end-to-end.

Contribution

It proposes a Grassmannian learning mutual subspace method (G-LMSM) that embeds subspace learning into CNNs for more effective image set classification.

Findings

Effective on hand shape recognition tasks

Improves face identification accuracy

Stable and efficient training with Riemannian optimization

Abstract

This paper addresses the problem of object recognition given a set of images as input (e.g., multiple camera sources and video frames). Convolutional neural network (CNN)-based frameworks do not exploit these sets effectively, processing a pattern as observed, not capturing the underlying feature distribution as it does not consider the variance of images in the set. To address this issue, we propose the Grassmannian learning mutual subspace method (G-LMSM), a NN layer embedded on top of CNNs as a classifier, that can process image sets more effectively and can be trained in an end-to-end manner. The image set is represented by a low-dimensional input subspace; and this input subspace is matched with reference subspaces by a similarity of their canonical angles, an interpretable and easy to compute metric. The key idea of G-LMSM is that the reference subspaces are learned as points on the Grassmann manifold, optimized with Riemannian stochastic gradient descent. This learning is stable, efficient and theoretically well-grounded. We demonstrate the effectiveness of our proposed method on hand shape recognition, face identification, and facial emotion recognition.