SOSAE: Self-Organizing Sparse AutoEncoder

TL;DR

SOSAE introduces a self-organizing regularization technique for autoencoders that dynamically adjusts feature space size, significantly reducing computational cost while preserving data representation quality.

Contribution

It proposes a novel regularization method inspired by physics that enables autoencoders to automatically determine optimal feature space dimensionality.

Findings

Reduces FLOPs by up to 130 times compared to baselines

Maintains comparable performance and data representation quality

Automatically adapts feature space size during training

Abstract

The process of tuning the size of the hidden layers for autoencoders has the benefit of providing optimally compressed representations for the input data. However, such hyper-parameter tuning process would take a lot of computation and time effort with grid search as the default option. In this paper, we introduce the Self-Organization Regularization for Autoencoders that dynamically adapts the dimensionality of the feature space to the optimal size. Inspired by physics concepts, Self-Organizing Sparse AutoEncoder (SOSAE) induces sparsity in feature space in a structured way that permits the truncation of the non-active part of the feature vector without any loss of information. This is done by penalizing the autoencoder based on the magnitude and the positional index of the feature vector dimensions, which during training constricts the feature space in both terms. Extensive experiments on various datasets show that our SOSAE can tune the feature space dimensionality up to 130 times lesser Floating-point Operations (FLOPs) than other baselines while maintaining the same quality of tuning and performance.