Unsupervised Learning by Predicting Noise

TL;DR

This paper presents a novel unsupervised learning framework for deep neural networks that aligns features to fixed noise targets, avoiding collapse and scaling efficiently to large datasets.

Contribution

It introduces Noise As Targets (NAT), a domain-agnostic method that trains deep networks without supervision, outperforming existing unsupervised techniques on standard benchmarks.

Findings

Achieves competitive results on ImageNet and Pascal VOC

Scales efficiently to millions of images

Avoids trivial solutions and feature collapse

Abstract

Convolutional neural networks provide visual features that perform remarkably well in many computer vision applications. However, training these networks requires significant amounts of supervision. This paper introduces a generic framework to train deep networks, end-to-end, with no supervision. We propose to fix a set of target representations, called Noise As Targets (NAT), and to constrain the deep features to align to them. This domain agnostic approach avoids the standard unsupervised learning issues of trivial solutions and collapsing of features. Thanks to a stochastic batch reassignment strategy and a separable square loss function, it scales to millions of images. The proposed approach produces representations that perform on par with state-of-the-art unsupervised methods on ImageNet and Pascal VOC.