Autoencoding Slow Representations for Semi-supervised Data Efficient Regression

TL;DR

This paper introduces a novel slowness regularization for unsupervised representation learning in autoencoders, improving data efficiency and downstream regression performance by leveraging temporal similarity inspired by the brain's visual cortex.

Contribution

It proposes a new formulation of slowness regularization within the beta-VAE framework, compares existing and new slowness terms, and demonstrates their effectiveness in enhancing downstream task performance.

Findings

Slow representations improve downstream regression accuracy.

Slowness regularization enhances data efficiency in various domains.

FID can predict autoencoder performance in supervised tasks.

Abstract

The slowness principle is a concept inspired by the visual cortex of the brain. It postulates that the underlying generative factors of a quickly varying sensory signal change on a slower time scale. Unsupervised learning of intermediate representations utilizing abundant unlabeled sensory data can be leveraged to perform data-efficient supervised downstream regression. In this paper, we propose a general formulation of slowness for unsupervised representation learning adding a slowness regularization term to the estimate lower bound of the beta-VAE to encourage temporal similarity in observation and latent space. Within this framework we compare existing slowness regularization terms such as the L1 and L2 loss used in existing end-to-end methods, the SlowVAE and propose a new term based on Brownian motion. We empirically evaluate these slowness regularization terms with respect to their downstream task performance and data efficiency. We find that slow representations lead to equal or better downstream task performance and data efficiency in different experiment domains when compared to representations without slowness regularization. Finally, we discuss how the Frechet Inception Distance (FID), traditionally used to determine the generative capabilities of GANs, can serve as a measure to predict the performance of pre-trained Autoencoder model in a supervised downstream task and accelerate hyperparameter search.