Chernoff information of exponential families

TL;DR

This paper presents methods to compute or approximate Chernoff information for exponential family distributions, facilitating better error probability bounds in binary classification tasks.

Contribution

It provides a closed-form solution or an efficient approximation technique for Chernoff information within exponential families, leveraging geometric insights.

Findings

Closed-form expressions for Chernoff information in exponential families

Efficient geodesic bisection method for approximation

Enhanced bounds on Bayesian decision error probabilities

Abstract

Chernoff information upper bounds the probability of error of the optimal Bayesian decision rule for $2$-class classification problems. However, it turns out that in practice the Chernoff bound is hard to calculate or even approximate. In statistics, many usual distributions, such as Gaussians, Poissons or frequency histograms called multinomials, can be handled in the unified framework of exponential families. In this note, we prove that the Chernoff information for members of the same exponential family can be either derived analytically in closed form, or efficiently approximated using a simple geodesic bisection optimization technique based on an exact geometric characterization of the "Chernoff point" on the underlying statistical manifold.