Efficient Unsupervised Visual Representation Learning with Explicit Cluster Balancing

TL;DR

ExCB introduces an online cluster balancing method for self-supervised visual learning, improving scalability and resource efficiency while maintaining state-of-the-art performance across various dataset sizes.

Contribution

The paper presents ExCB, a novel online cluster balancing technique that enhances self-supervised learning by avoiding large batch requirements and scalability issues.

Findings

Achieves state-of-the-art results with fewer resources.

Effective even with small batch sizes.

Scalable to large datasets.

Abstract

Self-supervised learning has recently emerged as the preeminent pretraining paradigm across and between modalities, with remarkable results. In the image domain specifically, group (or cluster) discrimination has been one of the most successful methods. However, such frameworks need to guard against heavily imbalanced cluster assignments to prevent collapse to trivial solutions. Existing works typically solve this by reweighing cluster assignments to promote balance, or with offline operations (e.g. regular re-clustering) that prevent collapse. However, the former typically requires large batch sizes, which leads to increased resource requirements, and the latter introduces scalability issues with regard to large datasets. In this work, we propose ExCB, a framework that tackles this problem with a novel cluster balancing method. ExCB estimates the relative size of the clusters across batches and balances them by adjusting cluster assignments, proportionately to their relative size and in an online manner. Thereby, it overcomes previous methods' dependence on large batch sizes and is fully online, and therefore scalable to any dataset. We conduct extensive experiments to evaluate our approach and demonstrate that ExCB: a) achieves state-of-the-art results with significantly reduced resource requirements compared to previous works, b) is fully online, and therefore scalable to large datasets, and c) is stable and effective even with very small batch sizes.