Quantitative Estimation of Target Task Performance from Unsupervised Pretext Task in Semi/Self-Supervised Learning

TL;DR

This paper develops a theoretical framework and a low-cost estimation method to predict target task performance in semi/self-supervised learning based on assumptions about pretext tasks.

Contribution

It introduces a theory linking pretext task impact to learnability, reliability, and completeness, and proposes a method to estimate target performance before training.

Findings

Estimated performance correlates strongly with actual performance.

Benchmark of over 100 pretext tasks validates the estimation method.

The approach enables preemptive assessment of pretext task suitability.

Abstract

The effectiveness of unlabeled data in Semi/Self-Supervised Learning (SSL) depends on appropriate assumptions for specific scenarios, thereby enabling the selection of beneficial unsupervised pretext tasks. However, existing research has paid limited attention to assumptions in SSL, resulting in practical situations where the compatibility between the unsupervised pretext tasks and the target scenarios can only be assessed after training and validation. This paper centers on the assumptions underlying unsupervised pretext tasks and explores the feasibility of preemptively estimating the impact of unsupervised pretext tasks at low cost. Through rigorous derivation, we show that the impact of unsupervised pretext tasks on target performance depends on three factors: assumption learnability with respect to the model, assumption reliability with respect to the data, and assumption completeness with respect to the target. Building on this theory, we propose a low-cost estimation method that can quantitatively estimate the actual target performance. We build a benchmark of over one hundred pretext tasks and demonstrate that estimated performance strongly correlates with the actual performance obtained through large-scale training and validation.