Learning Downstream Task by Selectively Capturing Complementary Knowledge from Multiple Self-supervisedly Learning Pretexts

TL;DR

This paper introduces a method that adaptively combines complementary representations from multiple self-supervised pretexts using attention, improving downstream task performance especially with limited labeled data.

Contribution

It proposes a novel attention-based approach to selectively utilize diverse pretext representations for downstream tasks, supported by theoretical proof of the benefits of multiple pretexts.

Findings

Outperforms existing pretext-matching methods in knowledge gathering.

Effectively reduces negative transfer in downstream tasks.

Theoretically proves the advantage of using diverse pretexts.

Abstract

Self-supervised learning (SSL), as a newly emerging unsupervised representation learning paradigm, generally follows a two-stage learning pipeline: 1) learning invariant and discriminative representations with auto-annotation pretext(s), then 2) transferring the representations to assist downstream task(s). Such two stages are usually implemented separately, making the learned representation learned agnostic to the downstream tasks. Currently, most works are devoted to exploring the first stage. Whereas, it is less studied on how to learn downstream tasks with limited labeled data using the already learned representations. Especially, it is crucial and challenging to selectively utilize the complementary representations from diverse pretexts for a downstream task. In this paper, we technically propose a novel solution by leveraging the attention mechanism to adaptively squeeze suitable representations for the tasks. Meanwhile, resorting to information theory, we theoretically prove that gathering representation from diverse pretexts is more effective than a single one. Extensive experiments validate that our scheme significantly exceeds current popular pretext-matching based methods in gathering knowledge and relieving negative transfer in downstream tasks.