Distribution Matching for Self-Supervised Transfer Learning

TL;DR

This paper introduces Distribution Matching (DM), a self-supervised transfer learning method that aligns representation distributions with a reference, resulting in interpretable features and strong classification performance backed by theoretical guarantees.

Contribution

The paper presents a novel distribution matching approach for self-supervised transfer learning with theoretical analysis and competitive empirical results.

Findings

DM achieves competitive classification accuracy across multiple datasets.

Theoretical guarantees include population and sample theorems.

DM maintains interpretability of learned representations.

Abstract

In this paper, we propose a novel self-supervised transfer learning method called \underline{\textbf{D}}istribution \underline{\textbf{M}}atching (DM), which drives the representation distribution toward a predefined reference distribution while preserving augmentation invariance. DM results in a learned representation space that is intuitively structured and therefore easy to interpret. Experimental results across multiple real-world datasets and evaluation metrics demonstrate that DM performs competitively on target classification tasks compared to existing self-supervised transfer learning methods. Additionally, we provide robust theoretical guarantees for DM, including a population theorem and an end-to-end sample theorem. The population theorem bridges the gap between the self-supervised learning task and target classification accuracy, while the sample theorem shows that, even with a limited number of samples from the target domain, DM can deliver exceptional classification performance, provided the unlabeled sample size is sufficiently large.