Boarding for ISS: Imbalanced Self-Supervised: Discovery of a Scaled Autoencoder for Mixed Tabular Datasets

TL;DR

This paper addresses the challenge of imbalanced data in self-supervised learning for tabular datasets, proposing a novel balanced MSE metric for autoencoders that improves reconstruction quality in imbalanced scenarios.

Contribution

It introduces a Multi-Supervised Balanced MSE metric for autoencoders, specifically designed to handle imbalanced categorical variables in tabular data, improving learning outcomes.

Findings

Balanced MSE outperforms standard MSE on imbalanced datasets

Proposed metric reduces reconstruction error in imbalanced scenarios

Similar results to standard MSE when data is balanced

Abstract

The field of imbalanced self-supervised learning, especially in the context of tabular data, has not been extensively studied. Existing research has predominantly focused on image datasets. This paper aims to fill this gap by examining the specific challenges posed by data imbalance in self-supervised learning in the domain of tabular data, with a primary focus on autoencoders. Autoencoders are widely employed for learning and constructing a new representation of a dataset, particularly for dimensionality reduction. They are also often used for generative model learning, as seen in variational autoencoders. When dealing with mixed tabular data, qualitative variables are often encoded using a one-hot encoder with a standard loss function (MSE or Cross Entropy). In this paper, we analyze the drawbacks of this approach, especially when categorical variables are imbalanced. We propose a novel metric to balance learning: a Multi-Supervised Balanced MSE. This approach reduces the reconstruction error by balancing the influence of variables. Finally, we empirically demonstrate that this new metric, compared to the standard MSE: i) outperforms when the dataset is imbalanced, especially when the learning process is insufficient, and ii) provides similar results in the opposite case.