Maximum Entropy Linear Manifold for Learning Discriminative Low-dimensional Representation

TL;DR

This paper introduces MELM, a novel low-dimensional linear projection method that maximizes class discriminability, improving classification, visualization, and data analysis efficiency.

Contribution

MELM extends the Multithreshold Entropy Linear Classifier to find highly discriminative low-dimensional linear embeddings for classification and visualization.

Findings

Provides highly discriminative 2D data projections

Demonstrates invariance properties of the objective function

Establishes connections with PCA and error bounds

Abstract

Representation learning is currently a very hot topic in modern machine learning, mostly due to the great success of the deep learning methods. In particular low-dimensional representation which discriminates classes can not only enhance the classification procedure, but also make it faster, while contrary to the high-dimensional embeddings can be efficiently used for visual based exploratory data analysis. In this paper we propose Maximum Entropy Linear Manifold (MELM), a multidimensional generalization of Multithreshold Entropy Linear Classifier model which is able to find a low-dimensional linear data projection maximizing discriminativeness of projected classes. As a result we obtain a linear embedding which can be used for classification, class aware dimensionality reduction and data visualization. MELM provides highly discriminative 2D projections of the data which can be used as a method for constructing robust classifiers. We provide both empirical evaluation as well as some interesting theoretical properties of our objective function such us scale and affine transformation invariance, connections with PCA and bounding of the expected balanced accuracy error.