A New Approach of Deriving Bounds between Entropy and Error from Joint Distribution: Case Study for Binary Classifications

TL;DR

This paper introduces a novel method to derive bounds between entropy and error based on joint distributions, specifically applied to binary classification, providing exact and tighter bounds than existing ones.

Contribution

It proposes a new approach linking entropy-error bounds directly to joint distributions and derives closed-form bounds for binary classification errors.

Findings

Fano's lower bound is exact for all error types.

A new tighter upper bound for Bayesian error is established.

Closed-form bounds relate error probabilities to joint distributions.

Abstract

The existing upper and lower bounds between entropy and error are mostly derived through an inequality means without linking to joint distributions. In fact, from either theoretical or application viewpoint, there exists a need to achieve a complete set of interpretations to the bounds in relation to joint distributions. For this reason, in this work we propose a new approach of deriving the bounds between entropy and error from a joint distribution. The specific case study is given on binary classifications, which can justify the need of the proposed approach. Two basic types of classification errors are investigated, namely, the Bayesian and non-Bayesian errors. For both errors, we derive the closed-form expressions of upper bound and lower bound in relation to joint distributions. The solutions show that Fano's lower bound is an exact bound for any type of errors in a relation diagram of "Error Probability vs. Conditional Entropy". A new upper bound for the Bayesian error is derived with respect to the minimum prior probability, which is generally tighter than Kovalevskij's upper bound.