Towards Understanding Why FixMatch Generalizes Better Than Supervised Learning

TL;DR

This paper provides the first theoretical explanation for why FixMatch semi-supervised learning outperforms supervised learning in deep neural networks, highlighting differences in feature learning and proposing an improved method called SA-FixMatch.

Contribution

It offers a novel theoretical framework explaining FixMatch's superior generalization and introduces SA-FixMatch, an improved variant inspired by the analysis.

Findings

FixMatch learns all discriminative features of each class

Supervised learning captures only a subset of features due to lottery ticket hypothesis

SA-FixMatch shows improved generalization in experiments

Abstract

Semi-supervised learning (SSL), exemplified by FixMatch (Sohn et al., 2020), has shown significant generalization advantages over supervised learning (SL), particularly in the context of deep neural networks (DNNs). However, it is still unclear, from a theoretical standpoint, why FixMatch-like SSL algorithms generalize better than SL on DNNs. In this work, we present the first theoretical justification for the enhanced test accuracy observed in FixMatch-like SSL applied to DNNs by taking convolutional neural networks (CNNs) on classification tasks as an example. Our theoretical analysis reveals that the semantic feature learning processes in FixMatch and SL are rather different. In particular, FixMatch learns all the discriminative features of each semantic class, while SL only randomly captures a subset of features due to the well-known lottery ticket hypothesis. Furthermore, we show that our analysis framework can be applied to other FixMatch-like SSL methods, e.g., FlexMatch, FreeMatch, Dash, and SoftMatch. Inspired by our theoretical analysis, we develop an improved variant of FixMatch, termed Semantic-Aware FixMatch (SA-FixMatch). Experimental results corroborate our theoretical findings and the enhanced generalization capability of SA-FixMatch.