Generative vs. Discriminative modeling under the lens of uncertainty quantification

TL;DR

This paper compares generative and discriminative models in terms of uncertainty quantification, analyzing their ability to leverage information, handle imbalanced data, and support semi-supervised learning, with practical sampling insights.

Contribution

It provides a comprehensive comparison of generative and discriminative approaches focusing on uncertainty quantification and introduces a general sampling scheme for supervised and semi-supervised learning.

Findings

Generative models better incorporate prior information.

Discriminative models excel with balanced datasets.

Sampling schemes improve inference in both approaches.

Abstract

Learning a parametric model from a given dataset indeed enables to capture intrinsic dependencies between random variables via a parametric conditional probability distribution and in turn predict the value of a label variable given observed variables. In this paper, we undertake a comparative analysis of generative and discriminative approaches which differ in their construction and the structure of the underlying inference problem. Our objective is to compare the ability of both approaches to leverage information from various sources in an epistemic uncertainty aware inference via the posterior predictive distribution. We assess the role of a prior distribution, explicit in the generative case and implicit in the discriminative case, leading to a discussion about discriminative models suffering from imbalanced dataset. We next examine the double role played by the observed variables in the generative case, and discuss the compatibility of both approaches with semi-supervised learning. We also provide with practical insights and we examine how the modeling choice impacts the sampling from the posterior predictive distribution. With regard to this, we propose a general sampling scheme enabling supervised learning for both approaches, as well as semi-supervised learning when compatible with the considered modeling approach. Throughout this paper, we illustrate our arguments and conclusions using the example of affine regression, and validate our comparative analysis through classification simulations using neural network based models.