A Hybrid of Generative and Discriminative Models Based on the Gaussian-coupled Softmax Layer

TL;DR

This paper introduces a hybrid neural network model combining generative and discriminative features using a Gaussian-coupled softmax layer, enhancing semi-supervised learning and confidence calibration.

Contribution

The paper proposes a novel Gaussian-coupled softmax layer that integrates generative and discriminative modeling within a single neural network architecture.

Findings

Effective semi-supervised learning performance

Improved confidence calibration results

Successful integration of generative and discriminative features

Abstract

Generative models have advantageous characteristics for classification tasks such as the availability of unsupervised data and calibrated confidence, whereas discriminative models have advantages in terms of the simplicity of their model structures and learning algorithms and their ability to outperform their generative counterparts. In this paper, we propose a method to train a hybrid of discriminative and generative models in a single neural network (NN), which exhibits the characteristics of both models. The key idea is the Gaussian-coupled softmax layer, which is a fully connected layer with a softmax activation function coupled with Gaussian distributions. This layer can be embedded into an NN-based classifier and allows the classifier to estimate both the class posterior distribution and the class-conditional data distribution. We demonstrate that the proposed hybrid model can be applied to semi-supervised learning and confidence calibration.