The Hellinger Bounds on the Kullback-Leibler Divergence and the Bernstein Norm

TL;DR

This paper establishes necessary and sufficient conditions under which the Hellinger distance bounds the Kullback-Leibler divergence, variation, and Bernstein norm, generalizing previous results and aiding in the analysis of likelihood models.

Contribution

It provides a unified characterization of when the Hellinger distance bounds other discrepancy measures, accommodating unbounded likelihood ratios and extending prior results.

Findings

Characterizes conditions for Hellinger bounds on divergence measures

Generalizes previous bounds to unbounded likelihood ratios

Relaxes regularity conditions for sieve maximum likelihood estimators

Abstract

The Kullback-Leibler divergence, the Kullback-Leibler variation, and the Bernstein "norm" are used to quantify discrepancies among probability distributions in likelihood models such as nonparametric maximum likelihood and nonparametric Bayes. They are closely related to the Hellinger distance, which is often easier to work with. Consequently, it is of interest to characterize conditions under which the Hellinger distance serves as an upper bound for these measures. This article characterizes a necessary and sufficient condition for each of the discrepancy measures to be bounded by the Hellinger distance. It accommodates unbounded likelihood ratios and generalizes all previously known results. We then apply it to relax the regularity condition for the sieve maximum likelihood estimator.