Unsupervised Visual Representation Learning via Mutual Information Regularized Assignment

TL;DR

This paper introduces MIRA, an unsupervised learning method that uses mutual information maximization for pseudo-labeling, achieving state-of-the-art results on ImageNet with fewer training epochs.

Contribution

MIRA formulates pseudo-labeling as an optimization problem maximizing mutual information, providing a simple clustering-based approach without extra constraints or techniques.

Findings

Achieves 75.6% linear evaluation accuracy on ImageNet with ResNet-50 in 400 epochs.

Outperforms baseline methods on downstream tasks.

Converges to optimal pseudo-labels through fixed-point iteration.

Abstract

This paper proposes Mutual Information Regularized Assignment (MIRA), a pseudo-labeling algorithm for unsupervised representation learning inspired by information maximization. We formulate online pseudo-labeling as an optimization problem to find pseudo-labels that maximize the mutual information between the label and data while being close to a given model probability. We derive a fixed-point iteration method and prove its convergence to the optimal solution. In contrast to baselines, MIRA combined with pseudo-label prediction enables a simple yet effective clustering-based representation learning without incorporating extra training techniques or artificial constraints such as sampling strategy, equipartition constraints, etc. With relatively small training epochs, representation learned by MIRA achieves state-of-the-art performance on various downstream tasks, including the linear/k-NN evaluation and transfer learning. Especially, with only 400 epochs, our method applied to ImageNet dataset with ResNet-50 architecture achieves 75.6% linear evaluation accuracy.