Generative Max-Mahalanobis Classifiers for Image Classification, Generation and More

TL;DR

This paper introduces a Generative Max-Mahalanobis Classifier (GMMC) that models data explicitly for improved image classification, generation, calibration, and robustness, outperforming previous energy-based models like JEM.

Contribution

It proposes GMMC, a generative classifier based on the Max-Mahalanobis approach, capable of joint discriminative and generative training for enhanced image tasks.

Findings

GMMC achieves state-of-the-art discriminative performance.

GMMC outperforms JEM in calibration and robustness.

GMMC effectively handles image generation and out-of-distribution detection.

Abstract

Joint Energy-based Model (JEM) of Grathwohl et al. shows that a standard softmax classifier can be reinterpreted as an energy-based model (EBM) for the joint distribution p(x,y); the resulting model can be optimized to improve calibration, robustness, and out-of-distribution detection, while generating samples rivaling the quality of recent GAN-based approaches. However, the softmax classifier that JEM exploits is inherently discriminative and its latent feature space is not well formulated as probabilistic distributions, which may hinder its potential for image generation and incur training instability. We hypothesize that generative classifiers, such as Linear Discriminant Analysis (LDA), might be more suitable for image generation since generative classifiers model the data generation process explicitly. This paper therefore investigates an LDA classifier for image classification and generation. In particular, the Max-Mahalanobis Classifier (MMC), a special case of LDA, fits our goal very well. We show that our Generative MMC (GMMC) can be trained discriminatively, generatively, or jointly for image classification and generation. Extensive experiments on multiple datasets show that GMMC achieves state-of-the-art discriminative and generative performances, while outperforming JEM in calibration, adversarial robustness, and out-of-distribution detection by a significant margin. Our source code is available at https://github.com/sndnyang/GMMC.