Input and Weight Space Smoothing for Semi-supervised Learning

TL;DR

This paper introduces a novel semi-supervised learning regularization technique that smooths both input and weight spaces, improving performance with a simple architecture and minimal data augmentation.

Contribution

It presents a new method combining input-space and weight-space smoothing via an adversarial min-max optimization, enhancing semi-supervised learning.

Findings

Achieves state-of-the-art performance without heavy data augmentation.

Uses a novel adversarial block coordinate descent algorithm.

Demonstrates complementary effects of input and weight smoothing.

Abstract

We propose regularizing the empirical loss for semi-supervised learning by acting on both the input (data) space, and the weight (parameter) space. We show that the two are not equivalent, and in fact are complementary, one affecting the minimality of the resulting representation, the other insensitivity to nuisance variability. We propose a method to perform such smoothing, which combines known input-space smoothing with a novel weight-space smoothing, based on a min-max (adversarial) optimization. The resulting Adversarial Block Coordinate Descent (ABCD) algorithm performs gradient ascent with a small learning rate for a random subset of the weights, and standard gradient descent on the remaining weights in the same mini-batch. It achieves comparable performance to the state-of-the-art without resorting to heavy data augmentation, using a relatively simple architecture.