Learning Rich Nearest Neighbor Representations from Self-supervised Ensembles

TL;DR

This paper introduces a novel method for ensembling self-supervised models by learning representations through gradient descent at inference time, enhancing transfer learning performance and representation quality.

Contribution

It proposes a new framework for self-supervised model ensembling that directly learns representations via gradient descent during inference, improving transferability and representation quality.

Findings

Improves k-nearest neighbors performance on in-domain data.

Enhances transfer learning performance across datasets.

Single models also benefit from the proposed representation learning.

Abstract

Pretraining convolutional neural networks via self-supervision, and applying them in transfer learning, is an incredibly fast-growing field that is rapidly and iteratively improving performance across practically all image domains. Meanwhile, model ensembling is one of the most universally applicable techniques in supervised learning literature and practice, offering a simple solution to reliably improve performance. But how to optimally combine self-supervised models to maximize representation quality has largely remained unaddressed. In this work, we provide a framework to perform self-supervised model ensembling via a novel method of learning representations directly through gradient descent at inference time. This technique improves representation quality, as measured by k-nearest neighbors, both on the in-domain dataset and in the transfer setting, with models transferable from the former setting to the latter. Additionally, this direct learning of feature through backpropagation improves representations from even a single model, echoing the improvements found in self-distillation.