Enhancing Representation Learning on High-Dimensional, Small-Size Tabular Data: A Divide and Conquer Method with Ensembled VAEs

TL;DR

This paper introduces a novel divide-and-conquer ensemble VAE method that improves representation learning in high-dimensional, small-sample tabular data, enhancing downstream classification, disentanglement, and robustness.

Contribution

It proposes an ensemble of lightweight VAEs with a new joint posterior factorization for better HDLSS representation learning, data augmentation, and robustness.

Findings

Better latent representations in HDLSS settings

Higher accuracy in downstream classification

Improved disentanglement and robustness

Abstract

Variational Autoencoders and their many variants have displayed impressive ability to perform dimensionality reduction, often achieving state-of-the-art performance. Many current methods however, struggle to learn good representations in High Dimensional, Low Sample Size (HDLSS) tasks, which is an inherently challenging setting. We address this challenge by using an ensemble of lightweight VAEs to learn posteriors over subsets of the feature-space, which get aggregated into a joint posterior in a novel divide-and-conquer approach. Specifically, we present an alternative factorisation of the joint posterior that induces a form of implicit data augmentation that yields greater sample efficiency. Through a series of experiments on eight real-world datasets, we show that our method learns better latent representations in HDLSS settings, which leads to higher accuracy in a downstream classification task. Furthermore, we verify that our approach has a positive effect on disentanglement and achieves a lower estimated Total Correlation on learnt representations. Finally, we show that our approach is robust to partial features at inference, exhibiting little performance degradation even with most features missing.