Finding the Maximizers of the Information Divergence from an Exponential Family

TL;DR

This paper develops a new approach to find the probability measures that maximize the divergence from an exponential family, transforming the problem into a convex optimization task and providing algorithms to identify these maximizers.

Contribution

It introduces a novel method to characterize and compute the maximizers of divergence from exponential families, including algorithms and applications to previously unknown cases.

Findings

Transformation of the divergence maximization problem into a convex optimization problem.

Development of two algorithms to find divergence maximizers.

Application to examples where maximizers were previously unknown.

Abstract

This paper investigates maximizers of the information divergence from an exponential family $E$. It is shown that the $rI$-projection of a maximizer $P$ to $E$ is a convex combination of $P$ and a probability measure $P_-$ with disjoint support and the same value of the sufficient statistics $A$. This observation can be used to transform the original problem of maximizing $D(\cdot||E)$ over the set of all probability measures into the maximization of a function $\Dbar$ over a convex subset of $\ker A$. The global maximizers of both problems correspond to each other. Furthermore, finding all local maximizers of $\Dbar$ yields all local maximizers of $D(\cdot||E)$. This paper also proposes two algorithms to find the maximizers of $\Dbar$ and applies them to two examples, where the maximizers of $D(\cdot||E)$ were not known before.