Constrained Mean Shift Using Distant Yet Related Neighbors for Representation Learning

TL;DR

This paper introduces a constrained mean-shift method for representation learning that leverages distant yet semantically related neighbors, improving performance in self-supervised and semi-supervised learning with less training resources.

Contribution

It generalizes mean-shift by constraining neighbor selection to distant but semantically related samples, enhancing learning efficiency and accuracy.

Findings

Outperforms MSF in self-supervised learning with different augmentation constraints.

Outperforms PAWS in semi-supervised learning with fewer resources.

Effective use of distant neighbors with same pseudo-label improves representation quality.

Abstract

We are interested in representation learning in self-supervised, supervised, and semi-supervised settings. Some recent self-supervised learning methods like mean-shift (MSF) cluster images by pulling the embedding of a query image to be closer to its nearest neighbors (NNs). Since most NNs are close to the query by design, the averaging may not affect the embedding of the query much. On the other hand, far away NNs may not be semantically related to the query. We generalize the mean-shift idea by constraining the search space of NNs using another source of knowledge so that NNs are far from the query while still being semantically related. We show that our method (1) outperforms MSF in SSL setting when the constraint utilizes a different augmentation of an image from the previous epoch, and (2) outperforms PAWS in semi-supervised setting with less training resources when the constraint ensures that the NNs have the same pseudo-label as the query.