Reproducible, incremental representation learning with Rosetta VAE

TL;DR

The paper introduces Rosetta VAE, a method for reproducible, incremental representation learning that enables models to build on prior results and improve consistency across training runs in scientific data analysis.

Contribution

The paper presents R-VAE, a novel approach that uses post hoc clustering and anchor points to allow VAEs to incrementally learn and reproduce prior representations.

Findings

R-VAE reconstructs data as well as standard VAEs.

R-VAE outperforms β-VAE in recovering target latent spaces.

R-VAE significantly increases consistency across training runs.

Abstract

Variational autoencoders are among the most popular methods for distilling low-dimensional structure from high-dimensional data, making them increasingly valuable as tools for data exploration and scientific discovery. However, unlike typical machine learning problems in which a single model is trained once on a single large dataset, scientific workflows privilege learned features that are reproducible, portable across labs, and capable of incrementally adding new data. Ideally, methods used by different research groups should produce comparable results, even without sharing fully trained models or entire data sets. Here, we address this challenge by introducing the Rosetta VAE (R-VAE), a method of distilling previously learned representations and retraining new models to reproduce and build on prior results. The R-VAE uses post hoc clustering over the latent space of a fully-trained model to identify a small number of Rosetta Points (input, latent pairs) to serve as anchors for training future models. An adjustable hyperparameter, $\rho$, balances fidelity to the previously learned latent space against accommodation of new data. We demonstrate that the R-VAE reconstructs data as well as the VAE and $\beta$-VAE, outperforms both methods in recovery of a target latent space in a sequential training setting, and dramatically increases consistency of the learned representation across training runs.