Exploring Transferability of Self-Supervised Learning by Task Conflict Calibration

TL;DR

This paper investigates how self-supervised learning representations transfer across tasks, identifies task conflict as a barrier, and introduces a calibration method to improve transferability through task-level information and bi-level optimization.

Contribution

The paper proposes Task Conflict Calibration (TC$^2$), a novel method that models and alleviates task conflict in SSL to enhance transferability of learned representations.

Findings

TC$^2$ consistently improves transferability across multiple downstream tasks.

Modeling task conflict with causal factors enhances SSL transferability.

Bi-level optimization effectively integrates TC$^2$ into SSL training.

Abstract

In this paper, we explore the transferability of SSL by addressing two central questions: (i) what is the representation transferability of SSL, and (ii) how can we effectively model this transferability? Transferability is defined as the ability of a representation learned from one task to support the objective of another. Inspired by the meta-learning paradigm, we construct multiple SSL tasks within each training batch to support explicitly modeling transferability. Based on empirical evidence and causal analysis, we find that although introducing task-level information improves transferability, it is still hindered by task conflict. To address this issue, we propose a Task Conflict Calibration (TC$^2$) method to alleviate the impact of task conflict. Specifically, it first splits batches to create multiple SSL tasks, infusing task-level information. Next, it uses a factor extraction network to produce causal generative factors for all tasks and a weight extraction network to assign dedicated weights to each sample, employing data reconstruction, orthogonality, and sparsity to ensure effectiveness. Finally, TC$^2$ calibrates sample representations during SSL training and integrates into the pipeline via a two-stage bi-level optimization framework to boost the transferability of learned representations. Experimental results on multiple downstream tasks demonstrate that our method consistently improves the transferability of SSL models.