Numerous inequalities and related communications accompanying discrete divergence models in probability spaces

TL;DR

This paper explores inequalities in probability spaces to improve communication systems and information transfer efficiency.

Contribution

A general divergence model is introduced, extending information-theoretic inequalities and results.

Findings

Various inequalities for finite sequences of positive real numbers were built.

Important communications concerning positive real numbers and divergence models were derived.

The model has implications for designing reliable and secure communication systems.

Abstract

The analysis of inequalities aids in the formulation of optimal coding schemes that improve the rate of information transfer while reducing the probability of errors. This consequently implies that we have a major impact on having solid and secured correspondence systems for applications running from broadcast communications to information pressure. Information theory inequalities constitute a theoretical toolbox for designing channels that can overcome a challenging environment, allowing information to be held and communicated reliably and securely in a world that is becoming ever more interconnected. The paper introduces a general divergence model in general probability spaces and extends known information-theoretic inequalities and results based on variational models. We have built various inequalities for finite sequences of positive real numbers, the specific cases of which are important in information theory, especially in connection with several divergence models that remain in the literature. Additionally, we have derived certain other important communications concerning positive real numbers in relation to some divergence models.

Figures

Equations

Full text

Introduction

The well-known validity of coding theory allows for the examination of code combinations using discrete probabilistic entropic models and supports applications in many domains. Shannon [1] developed the concept of entropy within discrete probability spaces, forming the foundation of information theory. Shannon’s widely accepted view of probabilistic entropy [1] expanded the coding theory literature by introducing several entropic models. This established process laid the groundwork for a discrete entropic model with unexpectedly favourable features.

We have the notion that $[eqn]$ is the assemblage of all unconnected possibility distributions with nonnegative members and complete support on a set of cardinality n and $[eqn]$ . A non-degenerate possibility distribution $[eqn]$ is thought to be a probability distribution. In many cases, transactions must be delivered using discrete probability distributions, each of which contains a positive real value. Consequently, we define the following sets:

$$[eqn]$$

For every probability distribution $[eqn]$ , we give various current entropic models:

The Shannon [1] entropy:

$$[eqn]$$

The Renyi [2] entropy:

$$[eqn]$$

The Havrda-Charvat [3] entropy:

$$[eqn]$$

Huang and Zhang [4] provided a new interpretation of Shannon’s [1] mutual information, noting that it is often difficult to compute for uncountable sample spaces. As a result, the authors observed that the asymptotic modus operandi pedestal on Fisher information often provides spectacular estimates to such information, and they mirrored the estimates pedestal on particular divergence models in the probability spaces. Furthermore, the authors performed numerical repetition and demonstrated that their predicted mode of operation was incredibly wonderful, with increasing ease of application to a wide range of real and hypothetical issues. Furthermore, discrete entropy models have a wide range of applications. Sholehkerdar et al. [5] said that entropy-based measures are often used to evaluate objective picture fusion supremacy due to single layer entropy execution and an inconsequential parameter set. In their communication, the authors made available a hypothetical research of picture fusion quality measures based on Tsallis [6] entropy. The goal of this research was to evaluate if the chosen quality measure was capable of providing the necessary performances predicted from a perfect information-based image fusion eminence metric. To assess the Tsallis [6] excellence metric, the study results suggested the use of an image realisation model to generate a closed-form appearance for excellence, with weighted averaging used as a fusion process. Their results show that the Tsallis-based excellence measure deviates from expected performance in terms of response to signal-to-noise ratio discrepancy and the effect of entropy order on the measured excellence indicator. Furthermore, the writers conducted evaluations on genuine photographs, the results of which confirmed conformity with the hypothetical exploration. Lenormand et al. [7] discussed the applications of entropy-based models in the urban environment, noting that describing and quantifying spatial inequalities across the urban background remains a complex and mysterious task that has been aided by the increasing availability of large geolocated catalogues. The outcomes of their investigation showed that the entropy-based measure of spatial distribution is a useful indicator of a region’s socioeconomic status and related factors. Lu et al. [8] said that measures of expectedness in physiological indicators based on entropy metrics have been widely used in the presenting domains of medical valuation and clinical identification. The authors proposed a novel entropy-based pattern knowledge for valuing physiological indicators that combines single spectrum analysis and entropy metrics. Saraiva, P. [9] provided brief and spontaneous summary of Shannon’s [1] entropy, including chosen characteristics, and offered applications of the model in two contrasting viewpoints from its inception: biological variety and a unique research on student mobility. Zhang and Shi [10] emphasized that Shannon’s [1] entropy is a fundamental building block of information theory and an essential component of Machine Learning (ML) approaches. The authors created asymptotic holdings that do not need any conventions on the original propagation, and these features allow for interval estimation and statistical tests with full Shannon entropy. Elgawad et al. [11] applied Shannon’s [1] entropy to statistics, including order statistics and several known distributions. Stoyanov et al. [12] employed maximum entropy technique to develop a unique concept for the M-indeterminacy of disseminations on the positive half-line (Stieltjes case) and then derived some important conclusions. Furthermore, the authors proved how the maximum entropy is connected to the symmetry characteristics and M-indeterminacy Wang et al. [13] and others are among the pioneers who have publicly stated their commitment to studying entropic models. In the following, we discuss the essential perspective of the discrete inaccuracy model as attributed to Kerridge [14].

Assume that an experimenter says that the chance of the result of a random experiment is, despite the true probability being. Then, utilizing certain compelling postulates, Kerridge [14] proved that the inaccuracy of the aforementioned claim is dictated by the later numerical exterior:

$$[eqn]$$

Sathar et al. [15] explored the prior inaccuracy model and therefore presented nonparametric estimators for these models. The authors carefully explored the asymptotic behaviour of these estimators under varied appropriateness and reliability requirements. Additionally, the authors examined the performance of the projected estimators using the Monte-Carlo simulation technique. Nath [16] also suggested the non-additive inaccuracy model predetermined by the following manifestation:

$$[eqn]$$

where $[eqn]$ , $[eqn]$ , $[eqn]$ an integer.

Different instigators proposed novel inaccuracy models because of their application in statistics, coding theory, and other related subjects. Kapur [17], Molloy and Ford [18], and others have made contributions to the characterisation and implementation of inaccuracy models.

The appreciation of entropic models has been investigated as measures of the amount of information for a given probability distribution; it is customary to scrutinize such models in order to assess the amount of information shared between two probability distributions in the sense of how close the two distributions are to one another. Such measurements are known as distance models, and the perception of distance is one of the most essential and fundamental ideas in the various applications of information theory, which has substantial implications in a wide variety of mathematical sciences domains.

Another opinion on the importance of such metrics is that several efforts have been made to broaden the idea of distance in domains other than mathematics. Disciplines such as economics, sociology, psychology, linguistics, genetics, and biology may all benefit from distance measurement. However, distance in such circumstances is not always geometrical, hence there is a need for modification when it comes to distance in probability spaces. Here, we emphasize the need of making adjustments while considering the distance model in probability space. To overcome these limitations, several novel divergence metrics have been proposed and defined. The most important and practically constructive divergence model is attributed to Kullback and Leibler [19], as defined by the following formula.:

$$[eqn]$$

The following generalized parametric models of directed divergence and equipped them with applications in the disciplines of statistics and operations research.

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ is a real parameter.

$$[eqn]$$

Parkash and Kakkar [20] developed the following extended parametric models of directed divergence and used them in the area of coding theory.

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

Pronzato et al. [21], Kumari and Sharma [22], Torra et al. [23], Khalaj et al. [24], Dwivedi et al. [25], Nielsen [26], Fukumizu [27], Li et al. [28], and others have all contributed to the suggestions, characterizations, generalizations, and implementations of divergence models. In the sequel, we illustrate some novel implications of divergence models. To satisfy our aim, we need the following standard findings from Ash [29] and Cover and Thomas [30]:

Research gap and Motivation

Although many divergence measures — especially the Kullback-Leibler and Rényi divergences — have received a great deal of attention, the current work tends to focus along the lines of properties or special cases of the divergences rather than an unified perspective on their relations through inequalities. The lack of cohesive inequalities linking these variables constrains their comparative analysis and practical interpretation across various probability models. Driven by this need, the current study formulates novel inequalities that generalize and expand existing divergence functions, hence providing enhanced understanding of the architecture of discrete probability spaces. This paper delineates new correlations among generalized divergence measures, offers a formal demonstration of the monotonic behaviour of Rényi divergence via inequality (2.7), and presents a symmetric divergence measure $[eqn]$ with enhanced analytical features. These findings jointly augment the theoretical foundation of divergence-based modelling and bolster its prospective applications in communication theory, coding systems, and statistical inference.

The impetus for creating new inequalities among these divergence metrics stems from their capacity to provide more precise limits and enhanced performance estimates for practical applications, including source coding, sensor data fusion, and statistical inference. Establishing these links connects abstract mathematical analysis to measurable improvements in practical systems.

Unless otherwise stated, throughout the next portions of the article, We will assume that $[eqn]$ $[eqn]$ $[eqn]$ and $[eqn]$ and are positive real values.

Theorem 1.1. With the above professed conventions, the subsequent inequality grips:

$$[eqn]$$

for all integers $[eqn]$ . If $[eqn]$ , then (1.13) holds only as an equality.

Theorem 1.2. With the above professed conventions and for $[eqn]$ if $[eqn]$ , then we attain the successive in equation:

$$[eqn]$$

with the sign of equality iff $[eqn]$ .

Theorem 1.3. For $[eqn]$ , $[eqn]$ , the subsequent inequality holds good:

$$[eqn]$$

for $[eqn]$ , $[eqn]$ and (1.15) holds except when $[eqn]$ .

Theorem 1.4. With the above professed conventions and for $[eqn]$ , the succeeding inequalities grip:

$$[eqn]$$

and

$$[eqn]$$

except when $[eqn]$ .

Theorem 1.5. With the above professed conventions and for $[eqn]$ a given integer, the subsequent inequality clutches good:

$$[eqn]$$

If $[eqn]$ , then (1.18) clutches as an equivalence. If $[eqn]$ , then the insignia of equivalence in (1.18) grips only for the equivalence in $[eqn]$

Various inequalities and other communications associated with divergence models

In this subdivision, we derive some inequalities and other communications concerning positive real numbers and deliberate their significance in relation to some divergence models.

Let $[eqn]$ , $[eqn]$ , $[eqn]$ an integer.

Definition 2.1. A function $[eqn]$ , $[eqn]$ an integer, is supposed to be a divergence function if, for all $[eqn]$ , $[eqn]$ ,

$$[eqn]$$

and

$$[eqn]$$

If, in addition, we have the subsequent manifestation:

$$[eqn]$$

then $[eqn]$ is said to be a symmetric divergence function.

Example 2.2. Choose $[eqn]$

where $[eqn]$ , $[eqn]$ an integer; is demarcated by the subsequent expression:

$$[eqn]$$

This divergence function is acknowledged as Kullback-Liebler’s [19] divergence function.

Choosing $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ an integer. If $[eqn]$ , then $[eqn]$ . If $[eqn]$ , then by using Theorem 1.1, it follows that $[eqn]$ with the insignia of equivalence iff $[eqn]$ .

Consequently, $[eqn]$

For $[eqn]$ , it has numerous interesting interpretations: Firstly, it is acknowledged as the distance of one probability distribution $[eqn]$ from another probability distribution $[eqn]$ . Secondly, it is interpreted as a measure of inaccuracy of error due to Kerridge [14]. Thirdly, it is interpreted as a measure of inefficiency due to Cover and Thomas [30] of assuming the probability distribution $[eqn]$ when, in representativeness, the true probability distribution is $[eqn]$ . Fourthly, it is likewise entitled the Shannon’s [1] information gain when the probability distribution $[eqn]$ is replaced by the probability distribution $[eqn]$ .

Definition 2.3. With the above professed conventions and for $[eqn]$ an integer, if $[eqn]$ , then the directed divergence denoted by $[eqn]$ of the sequence $[eqn]$ from the sequence $[eqn]$ is demarcated by the succeeding manifestation:

$$[eqn]$$

Making practice of Theorem 1.2, it shadows that

$[eqn]$ with the emblem of equivalence iff $[eqn]$ .

Accordingly $[eqn]$ appears to be an appropriate generalization of $[eqn]$ when $[eqn]$ , $[eqn]$ .

Now, Renyi ‘s [2] divergence models is specified by the succeeding expression:

$$[eqn]$$

The measure $[eqn]$ is entitled as the information gain of order $[eqn]$ , $[eqn]$ , $[eqn]$ , when the probability distribution $[eqn]$ is replaced by the probability distribution $[eqn]$ , $[eqn]$ an integer. We observe that

$[eqn]$ which is an expression termed as Shannon’s [1] information gain.

Consequently, $[eqn]$ may possibly be regarded as the information gain of order 1 and accordingly, it may be written as $[eqn]$ depending upon the situation.

Renyi [2] investigated and ascertained that $[eqn]$ with the emblem of equivalence iff $[eqn]$ $[eqn]$ an integer. We contribute with an alternative proof of this consequence. Keeping in understanding equation (1.15), it is sufficient to confirm that $[eqn]$ iff $[eqn]$ .

If $[eqn]$ $[eqn]$ ; then $[eqn]$ .

Now suppose that $[eqn]$ . Then $[eqn]$ .

Also, by Theorem 1.1, $[eqn]$ iff $[eqn]$ , that is, $[eqn]$ , as $[eqn]$ , $[eqn]$ .

Since $[eqn]$ and $[eqn]$ $[eqn]$ $[eqn]$ it follows that $[eqn]$ . Hence $[eqn]$ .

Proposition 2.4. Let $[eqn]$ , $[eqn]$ be a specified real constant and with the above professed conventions, for $[eqn]$ an integer, the succeeding inequality is continuously accurate:

$$[eqn]$$

If $[eqn]$ , then (2.7) holds as an equality. If $[eqn]$ , then the emblem of equivalence in (2.7) holds iff $[eqn]$ .

Proof. If $[eqn]$ , then the insignia of equivalence clamps in (2.7) as both sides of it reduce to $[eqn]$ .

Now consider $[eqn]$ . In this case, by Theorem 2.7, we acquire

$$[eqn]$$

with the insignia of equivalence in (2.8) iff $[eqn]$ or equivalently, $[eqn]$ as $[eqn]$ , $[eqn]$ . The inequality (2.7) follows from (2.8).

Now, we deliver arguments for the importance of (2.7) in the field of information theory.

For all $[eqn]$ $[eqn]$ $[eqn]$ a prearranged integer, we demonstrate that $[eqn]$ is a non-decreasing function of $[eqn]$ , $[eqn]$ $[eqn]$ . Indeed,

$$[eqn]$$

If in equation (2.7), we take $[eqn]$ $[eqn]$ $[eqn]$ such that $[eqn]$ $[eqn]$ then the term within brackets on the right hand side of (2.9) is a nonnegative real number.

Consequently, $[eqn]$ . Hence, $[eqn]$ is a non-decreasing function of $[eqn]$ .

If there happen indices $[eqn]$ and $[eqn]$ , such that $[eqn]$ , then strict inequality holds in (2.7). Consequently, if $[eqn]$ for some indices $[eqn]$ and $[eqn]$ , $[eqn]$ , then the term within brackets on the right hand side of (2.9) is a positive real number.

Accordingly, $[eqn]$ . Hence $[eqn]$ is a strictly monotonic increasing function of $[eqn]$ and as usual, it can certainly be ascertained that

$$[eqn]$$

and

$$[eqn]$$

Definition 2.5. Through the above professed conventions and for $[eqn]$ an integer, if the inequality $[eqn]$ clutches good, then the directed divergence of order $[eqn]$ $[eqn]$ , denoted by $[eqn]$ of the sequence $[eqn]$ from $[eqn]$ is demarcated subsequently:

$$[eqn]$$

Consider the case $[eqn]$ . In this case, by assumption, $[eqn]$ . Now from equation (2.12), we acquire the subsequent communication:

$[eqn]$ with the insignia of equivalence iff $[eqn]$

Suppose $[eqn]$ . Making practice of the conjecture $[eqn]$ , the equations (2.25) and (2.26), we achieve the succeeding manifestation:

$[eqn]$ unless $[eqn]$

Now we demonstrate that $[eqn]$ iff $[eqn]$ .

If $[eqn]$ , then $[eqn]$ follows from (4.12).

Now suppose $[eqn]$ . Replacing $[eqn]$ by $[eqn]$ in (1.18) and using the fact that $[eqn]$ , we obtain the subsequent appearance:

$$[eqn]$$

Consider the case $[eqn]$ . Upon employing the equations (2.5), (2.11), (2.12), (2.13)and the assumption $[eqn]$ we acquire the consequent exterior:

$$[eqn]$$

Since $[eqn]$ , equation (2.14) contributes with the subsequent appearance:

$[eqn]$ and $[eqn]$ .

Hence $[eqn]$ and $[eqn]$ . Consequently $[eqn]$

The case $[eqn]$ can be discussed similarly. Consequently, we have demonstrated that $[eqn]$ with the insignia of impartiality iff $[eqn]$

It is observed that all the measures $[eqn]$ $[eqn]$ $[eqn]$ and $[eqn]$ $[eqn]$ $[eqn]$ are additive.

Applications to shannon entropy and inaccuracy measures

Let us write

$$[eqn]$$

Motivated by this development, we may express some more measures of directed divergence Consider

$$[eqn]$$

where $[eqn]$ $[eqn]$ Since the right hand side of (2.16) may be a negative real number, $[eqn]$ does not seem to be a satisfactory measure of directed divergence.

Now, consider the subsequent manifestation:

$$[eqn]$$

where $[eqn]$ , $[eqn]$ . Here, too, the right hand side of (2.17) may be a negative real number and consequently, $[eqn]$ also does not seem to be satisfactory measure of directed divergence.

Let $[eqn]$ , $[eqn]$ a given integer be a measure of directed divergence.

Define $[eqn]$ as

$$[eqn]$$

Then $[eqn]$ is a symmetric measure of directed divergence satisfying (2.1), (2.2) and (2.3).

Now define $[eqn]$ as

$$[eqn]$$

Then $[eqn]$ is also a symmetric measure of directed divergence between the probability distributions $[eqn]$ , $[eqn]$ . An important situation arises if we choose $[eqn]$ where $[eqn]$ is the Kullback-Liebler’s [19] divergence defined by (2.4). In this case, writing $[eqn]$ in place of $[eqn]$ , we have

Symmetric divergence measures

$$[eqn]$$

The symmetric measure $[eqn]$ is usually called J-divergence [31]. It can be written as

$$[eqn]$$

Making use of (2.15), the J-divergence $[eqn]$ can be written as

$$[eqn]$$

For the sake of notational convenience, we write $[eqn]$ as $[eqn]$ and also write (2.21) as

$$[eqn]$$

Corresponding to (2.22), Nath [16,32] developed the subsequent manifestation:

$$[eqn]$$

where $[eqn]$ , $[eqn]$ . If $[eqn]$ , then $[eqn]$ . Now consider $[eqn]$ .

It is obvious that $[eqn]$ unless $[eqn]$ , $[eqn]$ .

If $[eqn]$ , $[eqn]$ ; then $[eqn]$ .

Consequently, $[eqn]$

Now suppose $[eqn]$ . We prove that $[eqn]$ , $[eqn]$ . The case $[eqn]$ is trivial. We restrict our discussion to the case when $[eqn]$ . In this case,

$$[eqn]$$

It stretches the subsequent communication:

$$[eqn]$$

Accordingly, we must have $[eqn]$ .

But $[eqn]$ . Consequently, $[eqn]$ . Since the right hand side of (2.23) is symmetric in $[eqn]$ and $[eqn]$ ; it follows that

$$[eqn]$$

Hence $[eqn]$ is a symmetric measure of directed divergence between the probability distributions $[eqn]$ , $[eqn]$ .

Practical example: (Application to image fusion — numerical illustration)

Consider a simple image-fusion scenario where two source images ( $[eqn]$ ) and ( $[eqn]$ ) produce discrete pixel-intensity histograms that are modelled by probability vectors

$$[eqn]$$ $$[eqn]$$

Here P and Q approximate pixel intensity distributions from different sensors or bands. We compute standard divergence measures between P and Q to illustrate how the inequalities and monotonicity results in this paper apply in practice.

1. Kullback–Leibler divergence (natural logarithm, “nats”):

$$[eqn]$$

Numerical values (computed term-by-term) give

$$[eqn]$$

hence the symmetric J-divergence

$$[eqn]$$

2. Rényi divergence of order $[eqn]$ (we use the conventional definition)

$$[eqn]$$ $$[eqn]$$

Monotonicity check and interpretation

For these P and Q we observe

$$[eqn]$$

which illustrates the monotonicity of the Rényi divergence in $[eqn]$ (the inequalities derived in the text). In the image-fusion context a small divergence (all values are small here) indicates that the fused image will retain characteristics of the original images with relatively little information loss; thus the inequalities and bounds we derived provide a quantitative basis for comparing fusion strategies and for choosing parameters in fusion algorithms.

Per-bin computation (illustrative terms)

Selected per-bin terms (rounded) used in the KL and Rényi computations:

**: **

These per-bin terms sum to the KL and the internal sums used in Rényi computations; shown here to document and reproduce the numbers

To continue further, our prerequisite is the subsequent consequence:

Result 2.9. If $[eqn]$ and $[eqn]$ be positive and unequal real numbers, then the succeeding inequalities are continuously correct:

$$[eqn]$$ $$[eqn]$$

The insignia of egalitarianism in, (2.24) and (2.25), grips when $[eqn]$ or $[eqn]$ or $[eqn]$ .

Corresponding to (2.22), let us contemplate the succeeding communication:

$$[eqn]$$

where $[eqn]$ , $[eqn]$ .

Making practice of (1.3) and (1.5), (1.17), we acquire the succeeding manifestation:

$$[eqn]$$

where $[eqn]$ , $[eqn]$ . If $[eqn]$ , then (2.27) gives $[eqn]$ . Now suppose $[eqn]$ . Consider any index $[eqn]$ , $[eqn]$ . If $[eqn]$ , then

$[eqn]$ for all $[eqn]$ , $[eqn]$ . If $[eqn]$ , $[eqn]$ , then by equations (2.24) and (2.25), we achieve the subsequent inequations:

$[eqn]$ if $[eqn]$

$[eqn]$ if $[eqn]$ .

Henceforth, we acquire the succeeding communications:

$[eqn]$ if $[eqn]$

$[eqn]$ if $[eqn]$ .

Also $[eqn]$ if $[eqn]$ and $[eqn]$ if $[eqn]$ .

Consequently, we acquire

$[eqn]$ for all $[eqn]$ , $[eqn]$ .

Thus, for any index $[eqn]$ , $[eqn]$ , $[eqn]$ for all $[eqn]$ , $[eqn]$

Therefore $[eqn]$ this corroborates the consequence.

Now we prove that $[eqn]$ iff $[eqn]$ . Indeed, from the above comprehensive conversation, it is obvious that if $[eqn]$ ; then $[eqn]$ . Now suppose that $[eqn]$ . We shall prove that $[eqn]$ . The case $[eqn]$ is trivial.

Now, we consider $[eqn]$ . From the above wide-spread arguments, it is clear that $[eqn]$ , is possible when $[eqn]$ , that is, $[eqn]$ . Making use of Result 2.9, this is possible only when $[eqn]$ , as $[eqn]$ and $[eqn]$ . Since the right hand side of (2.27) is symmetric in $[eqn]$ and $[eqn]$ ; it follows that

$$[eqn]$$

Consequently, for all $[eqn]$ , $[eqn]$ , $[eqn]$ is a symmetric measure of directed divergence.

Conclusion

Inequalities in information theory are supposed to be superfluous pillars in the realm of communication and data dispensation. Their prominence lies not only in their mathematical sophistication but likewise in their real-world solicitations across miscellaneous disciplines. As we traverse a period of speedy technological progression and accumulative dependence on interconnected classifications, inequalities in information theory make available a compass for manipulative communication infrastructures that can withstand the complexities of the contemporary world. By addressing the encounters pretended by noise, restricted bandwidth, and security apprehensions; these inequalities pave the approach for tough and proficient information conversation, eventually influencing the forthcoming landscape of communication machineries. The inequalities, theorems, definitions, and divergence-related communications presented in this study are useful in the field of information theory. All inequalities and related communications those are successful for positive real discovered by commissioning more discrete divergence models.