Self-Organizing Visual Prototypes for Non-Parametric Representation Learning

TL;DR

This paper introduces Self-Organizing Visual Prototypes (SOP), a novel unsupervised learning approach that uses multiple support embeddings per prototype to improve visual feature representation and achieve state-of-the-art results.

Contribution

The paper proposes a new SOP strategy with non-parametric adaptations and a SOP-MIM task, enhancing unsupervised visual feature learning with multiple local support embeddings.

Findings

Achieves state-of-the-art retrieval performance.

Supports increasing performance with more complex encoders.

Demonstrates effectiveness across multiple benchmarks.

Abstract

We present Self-Organizing Visual Prototypes (SOP), a new training technique for unsupervised visual feature learning. Unlike existing prototypical self-supervised learning (SSL) methods that rely on a single prototype to encode all relevant features of a hidden cluster in the data, we propose the SOP strategy. In this strategy, a prototype is represented by many semantically similar representations, or support embeddings (SEs), each containing a complementary set of features that together better characterize their region in space and maximize training performance. We reaffirm the feasibility of non-parametric SSL by introducing novel non-parametric adaptations of two loss functions that implement the SOP strategy. Notably, we introduce the SOP Masked Image Modeling (SOP-MIM) task, where masked representations are reconstructed from the perspective of multiple non-parametric local SEs. We comprehensively evaluate the representations learned using the SOP strategy on a range of benchmarks, including retrieval, linear evaluation, fine-tuning, and object detection. Our pre-trained encoders achieve state-of-the-art performance on many retrieval benchmarks and demonstrate increasing performance gains with more complex encoders.