On the Upper Bound of the Kullback-Leibler Divergence and Cross Entropy

TL;DR

This paper discusses the limitations of the unbounded Kullback-Leibler divergence, proposes a new bounded divergence measure, and compares it with other bounded alternatives, motivated by cost-benefit considerations.

Contribution

It introduces a new divergence measure that is bounded, addressing the unbounded nature of KL-divergence in certain applications.

Findings

Confirmed the need for bounded divergence in cost-benefit analysis

Proposed a new bounded divergence measure

Compared the new measure with existing bounded divergences

Abstract

This archiving article consists of several short reports on the discussions between the two authors over the past two years at Oxford and Madrid, and their work carried out during that period on the upper bound of the Kullback-Leibler divergence and cross entropy. The work was motivated by the cost-benefit ratio proposed by Chen and Golan [1], and the less desirable property that the Kullback-Leibler (KL) divergence used in the measure is unbounded. The work subsequently (i) confirmed that the KL-divergence used in the cost-benefit ratio should exhibit a bounded property, (ii) proposed a new divergence measure, and (iii) compared this new divergence measure with a few other bounded measures.