Perturbation of Deep Autoencoder Weights for Model Compression and Classification of Tabular Data

TL;DR

This paper introduces a weight perturbation method during pretraining of deep autoencoders that improves model compression and classification performance on tabular data, outperforming dropout in many cases.

Contribution

It proposes a novel weight perturbation routine during autoencoder pretraining that enhances model sparsity and classification accuracy over traditional regularization methods.

Findings

Weight perturbation achieves 15-40% sparsity in models.

Outperforms dropout in 4 out of 6 datasets.

Deep autoencoders with perturbation outperform traditional ML on correlated data.

Abstract

Fully connected deep neural networks (DNN) often include redundant weights leading to overfitting and high memory requirements. Additionally, the performance of DNN is often challenged by traditional machine learning models in tabular data classification. In this paper, we propose periodical perturbations (prune and regrow) of DNN weights, especially at the self-supervised pre-training stage of deep autoencoders. The proposed weight perturbation strategy outperforms dropout learning in four out of six tabular data sets in downstream classification tasks. The L1 or L2 regularization of weights at the same pretraining stage results in inferior classification performance compared to dropout or our weight perturbation routine. Unlike dropout learning, the proposed weight perturbation routine additionally achieves 15% to 40% sparsity across six tabular data sets for the compression of deep pretrained models. Our experiments reveal that a pretrained deep autoencoder with weight perturbation or dropout can outperform traditional machine learning in tabular data classification when fully connected DNN fails miserably. However, traditional machine learning models appear superior to any deep models when a tabular data set contains uncorrelated variables. Therefore, the success of deep models can be attributed to the inevitable presence of correlated variables in real-world data sets.