Criterion for the resemblance between the mother and the model distribution

TL;DR

This paper introduces a practical criterion based on Hellinger distance for assessing the similarity between a model distribution and a true, hidden distribution, especially useful for complex models like deep learning where density functions are unavailable.

Contribution

It proposes a novel, threshold-based criterion that does not require the model's probability density function and can be directly computed from sample sets, improving over traditional hypothesis tests.

Findings

The criterion is derived from the Bayes error rate.

An asymptotic bias of the estimator is established.

The criterion provides a positive conclusion when distributions are sufficiently close.

Abstract

If the probability distribution model aims to approximate the hidden mother distribution, it is imperative to establish a useful criterion for the resemblance between the mother and the model distributions. This study proposes a criterion that measures the Hellinger distance between discretized (quantized) samples from both distributions. Unlike information criteria such as AIC, this criterion does not require the probability density function of the model distribution, which cannot be explicitly obtained for a complicated model such as a deep learning machine. Second, it can draw a positive conclusion (i.e., both distributions are sufficiently close) under a given threshold, whereas a statistical hypothesis test, such as the Kolmogorov-Smirnov test, cannot genuinely lead to a positive conclusion when the hypothesis is accepted. In this study, we establish a reasonable threshold for the criterion deduced from the Bayes error rate and also present the asymptotic bias of the estimator of the criterion. From these results, a reasonable and easy-to-use criterion is established that can be directly calculated from the two sets of samples from both distributions.