Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models

TL;DR

This paper introduces ev-softmax, a sparse normalization function that preserves multimodality in high-dimensional generative models, enabling more efficient marginalization and training with probabilistic losses.

Contribution

The authors propose ev-softmax, a novel sparse normalization function with closed-form gradients and full support, improving multimodal distribution modeling in deep generative models.

Findings

Outperforms existing normalization techniques in distributional accuracy

Reduces dimensionality of probability distributions while maintaining multimodality

Compatible with probabilistic loss functions like NLL and KL divergence

Abstract

Many applications of generative models rely on the marginalization of their high-dimensional output probability distributions. Normalization functions that yield sparse probability distributions can make exact marginalization more computationally tractable. However, sparse normalization functions usually require alternative loss functions for training since the log-likelihood is undefined for sparse probability distributions. Furthermore, many sparse normalization functions often collapse the multimodality of distributions. In this work, we present $\textit{ev-softmax}$, a sparse normalization function that preserves the multimodality of probability distributions. We derive its properties, including its gradient in closed-form, and introduce a continuous family of approximations to $\textit{ev-softmax}$ that have full support and can be trained with probabilistic loss functions such as negative log-likelihood and Kullback-Leibler divergence. We evaluate our method on a variety of generative models, including variational autoencoders and auto-regressive architectures. Our method outperforms existing dense and sparse normalization techniques in distributional accuracy. We demonstrate that $\textit{ev-softmax}$ successfully reduces the dimensionality of probability distributions while maintaining multimodality.