Offline Clustering Approach to Self-supervised Learning for Class-imbalanced Image Data

TL;DR

This paper investigates how class-imbalance affects self-supervised learning and proposes an offline clustering method to improve pre-training on imbalanced image datasets, demonstrating potential performance gains.

Contribution

It introduces an offline clustering approach to enhance self-supervised pre-training on class-imbalanced data, addressing bias issues in representation learning.

Findings

Class-imbalance impacts self-supervised pre-training effectiveness.

Offline clustering of features can improve model performance on imbalanced data.

Knowledge distillation from expert models enhances overall learning.

Abstract

Class-imbalanced datasets are known to cause the problem of model being biased towards the majority classes. In this project, we set up two research questions: 1) when is the class-imbalance problem more prevalent in self-supervised pre-training? and 2) can offline clustering of feature representations help pre-training on class-imbalanced data? Our experiments investigate the former question by adjusting the degree of {\it class-imbalance} when training the baseline models, namely SimCLR and SimSiam on CIFAR-10 database. To answer the latter question, we train each expert model on each subset of the feature clusters. We then distill the knowledge of expert models into a single model, so that we will be able to compare the performance of this model to our baselines.