Meta-Learning Update Rules for Unsupervised Representation Learning

TL;DR

This paper introduces a meta-learned, biologically-inspired unsupervised learning rule that directly optimizes representations for semi-supervised classification, outperforming some existing methods and generalizing across architectures, datasets, and modalities.

Contribution

It proposes a novel meta-learning approach to develop an unsupervised update rule that is biologically plausible and effective for various neural network configurations and data types.

Findings

Meta-learned rule produces useful features for semi-supervised tasks.

Outperforms existing unsupervised learning techniques in some cases.

Generalizes across architectures, datasets, and modalities.

Abstract

A major goal of unsupervised learning is to discover data representations that are useful for subsequent tasks, without access to supervised labels during training. Typically, this involves minimizing a surrogate objective, such as the negative log likelihood of a generative model, with the hope that representations useful for subsequent tasks will arise as a side effect. In this work, we propose instead to directly target later desired tasks by meta-learning an unsupervised learning rule which leads to representations useful for those tasks. Specifically, we target semi-supervised classification performance, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations useful for this task. Additionally, we constrain our unsupervised update rule to a be a biologically-motivated, neuron-local function, which enables it to generalize to different neural network architectures, datasets, and data modalities. We show that the meta-learned update rule produces useful features and sometimes outperforms existing unsupervised learning techniques. We further show that the meta-learned unsupervised update rule generalizes to train networks with different widths, depths, and nonlinearities. It also generalizes to train on data with randomly permuted input dimensions and even generalizes from image datasets to a text task.