Stochastic Feature Mapping for PAC-Bayes Classification

TL;DR

This paper introduces a novel framework combining generative and discriminative models using PAC-Bayes risk bounds, leading to improved classification performance through an EM-like optimization process.

Contribution

It develops a unified approach coupling generative and discriminative models with stochastic feature mapping based on PAC-Bayes theory, enabling effective semi-supervised learning.

Findings

Achieves state-of-the-art classification accuracy.

Provides a feasible EM-like iterative optimization method.

Effectively couples generative and discriminative models.

Abstract

Probabilistic generative modeling of data distributions can potentially exploit hidden information which is useful for discriminative classification. This observation has motivated the development of approaches that couple generative and discriminative models for classification. In this paper, we propose a new approach to couple generative and discriminative models in an unified framework based on PAC-Bayes risk theory. We first derive the model-parameter-independent stochastic feature mapping from a practical MAP classifier operating on generative models. Then we construct a linear stochastic classifier equipped with the feature mapping, and derive the explicit PAC-Bayes risk bounds for such classifier for both supervised and semi-supervised learning. Minimizing the risk bound, using an EM-like iterative procedure, results in a new posterior over hidden variables (E-step) and the update rules of model parameters (M-step). The derivation of the posterior is always feasible due to the way of equipping feature mapping and the explicit form of bounding risk. The derived posterior allows the tuning of generative models and subsequently the feature mappings for better classification. The derived update rules of the model parameters are same to those of the uncoupled models as the feature mapping is model-parameter-independent. Our experiments show that the coupling between data modeling generative model and the discriminative classifier via a stochastic feature mapping in this framework leads to a general classification tool with state-of-the-art performance.