Unleash Model Potential: Bootstrapped Meta Self-supervised Learning

TL;DR

This paper introduces BMSSL, a novel framework combining self-supervised learning and meta-learning through bi-level optimization and bootstrapped targets, to learn robust visual representations from limited data without labels.

Contribution

The paper proposes a new Bootstrapped Meta Self-Supervised Learning framework that integrates meta-learning and self-supervised learning with a bi-level optimization and meta-gradient based bootstrapping.

Findings

Effective in learning from limited data without supervision

Achieves robustness and generalization in visual representations

Validated through comprehensive theoretical and empirical studies

Abstract

The long-term goal of machine learning is to learn general visual representations from a small amount of data without supervision, mimicking three advantages of human cognition: i) no need for labels, ii) robustness to data scarcity, and iii) learning from experience. Self-supervised learning and meta-learning are two promising techniques to achieve this goal, but they both only partially capture the advantages and fail to address all the problems. Self-supervised learning struggles to overcome the drawbacks of data scarcity, while ignoring prior knowledge that can facilitate learning and generalization. Meta-learning relies on supervised information and suffers from a bottleneck of insufficient learning. To address these issues, we propose a novel Bootstrapped Meta Self-Supervised Learning (BMSSL) framework that aims to simulate the human learning process. We first analyze the close relationship between meta-learning and self-supervised learning. Based on this insight, we reconstruct tasks to leverage the strengths of both paradigms, achieving advantages i and ii. Moreover, we employ a bi-level optimization framework that alternates between solving specific tasks with a learned ability (first level) and improving this ability (second level), attaining advantage iii. To fully harness its power, we introduce a bootstrapped target based on meta-gradient to make the model its own teacher. We validate the effectiveness of our approach with comprehensive theoretical and empirical study.