Reducing Data Complexity using Autoencoders with Class-informed Loss Functions

TL;DR

This paper introduces class-informed autoencoders that incorporate label information into the loss function to effectively reduce data complexity, improving feature learning for classification tasks.

Contribution

It proposes three novel autoencoder-based methods—Scorer, Skaler, and Slicer—that leverage class labels to enhance feature extraction for complex data.

Findings

Outperforms four popular unsupervised feature extraction techniques.

Effective in reducing data complexity for classification.

Validated on 27 datasets with diverse complexity metrics.

Abstract

Available data in machine learning applications is becoming increasingly complex, due to higher dimensionality and difficult classes. There exists a wide variety of approaches to measuring complexity of labeled data, according to class overlap, separability or boundary shapes, as well as group morphology. Many techniques can transform the data in order to find better features, but few focus on specifically reducing data complexity. Most data transformation methods mainly treat the dimensionality aspect, leaving aside the available information within class labels which can be useful when classes are somehow complex. This paper proposes an autoencoder-based approach to complexity reduction, using class labels in order to inform the loss function about the adequacy of the generated variables. This leads to three different new feature learners, Scorer, Skaler and Slicer. They are based on Fisher's discriminant ratio, the Kullback-Leibler divergence and least-squares support vector machines, respectively. They can be applied as a preprocessing stage for a binary classification problem. A thorough experimentation across a collection of 27 datasets and a range of complexity and classification metrics shows that class-informed autoencoders perform better than 4 other popular unsupervised feature extraction techniques, especially when the final objective is using the data for a classification task.