Viewmaker Networks: Learning Views for Unsupervised Representation Learning

TL;DR

Viewmaker networks are generative models that automatically learn to produce effective data views for unsupervised representation learning, reducing the need for manual design of data augmentations across various domains.

Contribution

We introduce viewmaker networks that learn to generate useful views for unsupervised learning, eliminating manual augmentation design and improving transfer performance across multiple data modalities.

Findings

Learned views enable comparable transfer accuracy to traditional augmentations on CIFAR-10.

Significantly outperform baseline augmentations on speech and sensor data.

Improve robustness to image corruptions when combined with handcrafted views.

Abstract

Many recent methods for unsupervised representation learning train models to be invariant to different "views," or distorted versions of an input. However, designing these views requires considerable trial and error by human experts, hindering widespread adoption of unsupervised representation learning methods across domains and modalities. To address this, we propose viewmaker networks: generative models that learn to produce useful views from a given input. Viewmakers are stochastic bounded adversaries: they produce views by generating and then adding an $\ell_p$-bounded perturbation to the input, and are trained adversarially with respect to the main encoder network. Remarkably, when pretraining on CIFAR-10, our learned views enable comparable transfer accuracy to the well-tuned SimCLR augmentations -- despite not including transformations like cropping or color jitter. Furthermore, our learned views significantly outperform baseline augmentations on speech recordings (+9% points, on average) and wearable sensor data (+17% points). Viewmakers can also be combined with handcrafted views: they improve robustness to common image corruptions and can increase transfer performance in cases where handcrafted views are less explored. These results suggest that viewmakers may provide a path towards more general representation learning algorithms -- reducing the domain expertise and effort needed to pretrain on a much wider set of domains. Code is available at https://github.com/alextamkin/viewmaker.