Don't Wait, Just Weight: Improving Unsupervised Representations by Learning Goal-Driven Instance Weights

TL;DR

BetaDataWeighter is a novel Bayesian instance weighting method that enhances self-supervised learning by prioritizing useful data and reducing training time, especially effective across different datasets.

Contribution

The paper introduces BetaDataWeighter, a Bayesian approach to learn instance weights that improve downstream accuracy and efficiency in self-supervised learning.

Findings

Achieves highest average accuracy across datasets.

Prunes up to 78% of data without accuracy loss.

Reduces training time by over 50%.

Abstract

In the absence of large labelled datasets, self-supervised learning techniques can boost performance by learning useful representations from unlabelled data, which is often more readily available. However, there is often a domain shift between the unlabelled collection and the downstream target problem data. We show that by learning Bayesian instance weights for the unlabelled data, we can improve the downstream classification accuracy by prioritising the most useful instances. Additionally, we show that the training time can be reduced by discarding unnecessary datapoints. Our method, BetaDataWeighter is evaluated using the popular self-supervised rotation prediction task on STL-10 and Visual Decathlon. We compare to related instance weighting schemes, both hand-designed heuristics and meta-learning, as well as conventional self-supervised learning. BetaDataWeighter achieves both the highest average accuracy and rank across datasets, and on STL-10 it prunes up to 78% of unlabelled images without significant loss in accuracy, corresponding to over 50% reduction in training time.