Unsupervised feature learning with discriminative encoder

TL;DR

This paper introduces DisCoder, an unsupervised discriminative model that learns useful features by jointly modeling data and latent variables, achieving state-of-the-art results without relying on labeled data.

Contribution

It proposes a novel EM-like training method with adversarial regularization for unsupervised feature learning using discriminative models.

Findings

Achieves state-of-the-art performance on several tasks.

Effectively learns both one-hot and real-valued latent features.

Demonstrates flexibility and robustness in unsupervised learning.

Abstract

In recent years, deep discriminative models have achieved extraordinary performance on supervised learning tasks, significantly outperforming their generative counterparts. However, their success relies on the presence of a large amount of labeled data. How can one use the same discriminative models for learning useful features in the absence of labels? We address this question in this paper, by jointly modeling the distribution of data and latent features in a manner that explicitly assigns zero probability to unobserved data. Rather than maximizing the marginal probability of observed data, we maximize the joint probability of the data and the latent features using a two step EM-like procedure. To prevent the model from overfitting to our initial selection of latent features, we use adversarial regularization. Depending on the task, we allow the latent features to be one-hot or real-valued vectors and define a suitable prior on the features. For instance, one-hot features correspond to class labels and are directly used for the unsupervised and semi-supervised classification task, whereas real-valued feature vectors are fed as input to simple classifiers for auxiliary supervised discrimination tasks. The proposed model, which we dub discriminative encoder (or DisCoder), is flexible in the type of latent features that it can capture. The proposed model achieves state-of-the-art performance on several challenging tasks.