Q-Match: Self-Supervised Learning by Matching Distributions Induced by a Queue

TL;DR

Q-Match is a self-supervised learning algorithm that leverages a queue of embeddings to match student-teacher distributions without labeled data, improving classification performance on tabular datasets.

Contribution

It introduces a novel distribution matching approach using a queue of embeddings, enabling effective self-supervised learning without labeled data.

Findings

Outperforms previous self-supervised methods on tabular data

Requires fewer labels for downstream tasks

Scales well with data size

Abstract

In semi-supervised learning, student-teacher distribution matching has been successful in improving performance of models using unlabeled data in conjunction with few labeled samples. In this paper, we aim to replicate that success in the self-supervised setup where we do not have access to any labeled data during pre-training. We introduce our algorithm, Q-Match, and show it is possible to induce the student-teacher distributions without any knowledge of downstream classes by using a queue of embeddings of samples from the unlabeled dataset. We focus our study on tabular datasets and show that Q-Match outperforms previous self-supervised learning techniques when measuring downstream classification performance. Furthermore, we show that our method is sample efficient--in terms of both the labels required for downstream training and the amount of unlabeled data required for pre-training--and scales well to the sizes of both the labeled and unlabeled data.