Approximate Maximum A Posteriori Inference with Entropic Priors

TL;DR

This paper introduces an iterative algorithm for MAP inference of multinomial distributions with entropic priors, promoting sparsity in applications like latent audio source decomposition where standard L1 penalties are unsuitable.

Contribution

It proposes a novel iterative method for MAP estimation using entropic priors, addressing the challenge of non-conjugacy with multinomial distributions.

Findings

Algorithm effectively promotes sparsity in multinomial parameters.

Applicable to latent source decomposition with sparse activations.

Provides a practical solution where L1 penalties are infeasible.

Abstract

In certain applications it is useful to fit multinomial distributions to observed data with a penalty term that encourages sparsity. For example, in probabilistic latent audio source decomposition one may wish to encode the assumption that only a few latent sources are active at any given time. The standard heuristic of applying an L1 penalty is not an option when fitting the parameters to a multinomial distribution, which are constrained to sum to 1. An alternative is to use a penalty term that encourages low-entropy solutions, which corresponds to maximum a posteriori (MAP) parameter estimation with an entropic prior. The lack of conjugacy between the entropic prior and the multinomial distribution complicates this approach. In this report I propose a simple iterative algorithm for MAP estimation of multinomial distributions with sparsity-inducing entropic priors.