An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training

TL;DR

This paper investigates energy-based models (EBMs) for domain-agnostic semi-supervised learning, comparing joint-training and pre-training approaches across image and language tasks, highlighting the superior performance of joint-training EBMs.

Contribution

It introduces the use of pre-training EBMs for SSL and provides a comprehensive experimental comparison with joint-training EBMs across multiple domains.

Findings

Joint-training EBMs outperform pre-training EBMs marginally.

Energy-based models are effective for domain-agnostic SSL.

Experiments cover image classification and natural language labeling.

Abstract

A class of recent semi-supervised learning (SSL) methods heavily rely on domain-specific data augmentations. In contrast, generative SSL methods involve unsupervised learning based on generative models by either joint-training or pre-training, and are more appealing from the perspective of being domain-agnostic, since they do not inherently require data augmentations. Joint-training estimates the joint distribution of observations and labels, while pre-training is taken over observations only. Recently, energy-based models (EBMs) have achieved promising results for generative modeling. Joint-training via EBMs for SSL has been explored with encouraging results across different data modalities. In this paper, we make two contributions. First, we explore pre-training via EBMs for SSL and compare it to joint-training. Second, a suite of experiments are conducted over domains of image classification and natural language labeling to give a realistic whole picture of the performances of EBM based SSL methods. It is found that joint-training EBMs outperform pre-training EBMs marginally but nearly consistently.