Comment on "R\'enyi entropy yields artifficial biases not in the data and incorrect updating due to the infinite-size data"

TL;DR

This paper critiques a previous claim that Rényi entropy violates Shore-Johnson axioms, clarifying misconceptions and demonstrating that, when properly analyzed, Rényi entropy remains consistent with these axioms.

Contribution

The authors identify and correct errors in prior reasoning, showing Rényi entropy does not violate Shore-Johnson axioms when properly evaluated.

Findings

Rényi entropy is consistent with Shore-Johnson axioms after correcting previous errors

Misinterpretations led to false claims of violations in earlier work

Proper analysis confirms Rényi entropy's suitability for inference

Abstract

In their recent paper [Phys. Rev. E 99 (2019) 032134], T. Oikinomou and B. Bagci have argued that R\'enyi entropy is ill-suited for inference purposes because it is not consistent with the Shore{ Johnson axioms of statistical estimation theory. In this Comment we seek to clarify the latter statement by showing that there are several issues in Oikinomou{Bagci reasonings which lead to erroneous conclusions. When all these issues are properly accounted for, no violation of Shore- Johnson axioms is found.

Equations

$$H_q(P)=\frac{1}{1-q}\ln \left(\sum _i{p}_i^q\right),q>0,$$

$$H_q(P||Q)=\frac{1}{q-1}\ln \left\{\int {\left[\frac{p(\mathbf{x})}{q(\mathbf{x})}\right]}^{q}q(\mathbf{x})\mathrm{d}\mathbf{x}\right\}.$$

$$H_q(P_{\alpha })>\alpha H_q(P_1)+\ge (1-\alpha )H_q(P_2)$$

$$=H_q(P_1)=H_q(P_2).$$

$$H_q(P||Q)=\frac{1}{q-1}\ln \left[\int {\left(\frac{dP}{dQ}\right)}^{q-1}\mathrm{d}P\right].$$

$$\frac{\partial }{\partial p_l}\left(\frac{\partial }{\partial p_k}-\frac{\partial }{\partial p_j}\right)\left[H-\alpha \sum _i{p}_i-\beta \sum _i{E}_i{p}_i\right]=0.$$

$$S(P)=f\left(\sum _{i}g(p_i)\right),$$

$$g^{\prime }(p_i)-\alpha -\beta E_i=0,$$

$$\alpha$$

$$\beta$$

$$p_i$$

$$C_f{g}^{\prime }(p_i)-{\alpha }_f-{\beta }_f{E}_i=0,$$

$$\text{mbox}{\alpha }_f=C_f\sum _i{p}_i{g}^{\prime }(p_i)-{\beta }_{f}E=C_f\alpha ,$$

$$\text{mbox}{\beta }_f=\frac{C_f{g}^{\prime }(p_i)-{\alpha }_f}{E_i}=C_f\beta ,$$

$$\text{mbox}{p}_i=(g^{\prime }{)}^{-1}\left(\frac{{\alpha }_f+{\beta }_f{E}_i}{C_f}\right)$$

$$\text{mbox}=(g^{\prime }{)}^{-1}\left(\alpha +\beta E_i\right),$$

$$\frac{\partial}{\partial p_{l}}\Big{\{}C_{f}(P)[g^{\prime}(p_{k})-g^{\prime}(p_{j})]\ +\ \beta(E_{k}-E_{j})\Big{\}},$$

$$\text{mbox}\frac{\partial C_f(P)}{\partial p_l}[g^{\prime }(p_k)-g^{\prime }(p_j)]$$

$$\displaystyle\mbox{\hskip 28.45274pt}+\ C_{f}(P)\frac{\partial}{\partial p_{l}}\Big{[}g^{\prime}(p_{k})-g^{\prime}(p_{j})\Big{]}\ =\ 0\,.$$

$$g^{\prime \prime \prime }(p_{ij})\frac{\partial p_{ij}}{\partial u_i}\frac{\partial p_{ij}}{\partial v_j}+g^{\prime \prime }(p_{ij})\frac{{\partial }^2{p}_{ij}}{\partial u_i\partial v_j}=0,$$

$$\frac{\partial p_{ij}}{\partial u_i}\frac{\partial p_{ij}}{\partial v_j}/\frac{{\partial }^2{p}_{ij}}{\partial u_i\partial v_j}=p_{ij}.$$

$$\frac{\partial p_{ij}}{\partial u_i}\frac{\partial p_{ij}}{\partial v_j}/\frac{{\partial }^2{p}_{ij}}{\partial u_i\partial v_j}=a{p}_{ij},$$

$$p_{ij}\propto {\left(u_i^{q-1}+v_j^{q-1}-1\right)}^{1/(q-1)}.$$

$$H_q(1,0,\dots ,0)\le H_q(P)\le H_q(1/n,\dots ,1/n),$$

$$\{1/n,\dots ,1/n\}\prec P\prec \{1,0,\dots ,0\}.$$

$$S_q^H(\rho )=\frac{{\left({\sum }_i{\rho }_i^{1/q}\right)}^{-q}-1}{1-q}.$$

Full text

Shore–Johnson axioms, Rényi Entropy, estimation theory

Introduction. —  Maximum entropy (MaxEnt) principle belongs among the most prominent concepts of contemporary statistical physics, information theory and statistical estimation. Its inception dates back to two seminal papers of E.T. Jaynes jaynes1 ; jaynes2 who first employed the Shannon information measure, or Shannon entropy (SE), in the framework of equilibrium statistical physics.

Over the years, Jaynes’ heuristic MaxEnt prescription has become a powerful instrument, e.g., in non-equilibrium statistical physics, astronomy, geophysics, biology, medical diagnosis or economics thurner-book ; kapur . The rationale behind this success is typically twofold: first, maximizing entropy minimizes the amount of prior information built into the distribution (i.e. MaxEnt distribution is maximally noncommittal with regard to missing information); second, many physical systems tend to move towards (or concentrate extremely close to) MaxEnt configurations over time Tsallis-book ; jaynes1 ; beck ; thurner-book .

With the advent of generalized entropies jizba3 ; tsallis2 ; havrda ; sharma ; Kaniadakis ; hanel ; korbel , a natural question has arisen as to whether the MaxEnt principle can be extended also to non-Shannonian entropies. This clearly cannot be decided within Jaynes’ heuristic framework — a sound mathematical qualification is needed. Since the MaxEnt principle is in its essence an inference method estimating the probability distributions from limited information, a pertinent mathematical basis should stem from theory of statistical estimation. Shore and Johnson (SJ) shore ; shoreII introduced a system of axioms, which ensure that the MaxEnt estimation procedure is consistent with desired properties of inference methods. These axioms are as follows shore ; shoreII :

uniqueness: the system should be unique; 2. 2.

permutation invariance: the permutation of states should not matter; 3. 3.

subset independence: It should not matter whether one treats disjoint subsets of system states in terms of separate conditional distributions or in terms of the full distribution; 4. 4.

system independence: It should not matter whether one accounts for independent constraints related to independent systems separately in terms of marginal distributions or in terms of full-system.

There is often also a fifth axiom, which was not included in the original systems of SJ axioms shore but appeared in later editions uffink :

maximality: In absence of any prior information, the uniform distribution should be the solution.

One can analogously define the set of axioms for the continuous systems with several adjustments shore ; uffink . First, for the continuous variables it is necessary to use the Minimum Relative Entropy (MinRel) principle, where the maximization of the entropy is replaced by minimization of the relative entropy subject to some given prior distribution. Second, the second axiom changes to coordinate invariance axiom, which states that the change of coordinate system should not matter. Third, the maximality axiom is replaced by no-information axiom in the way that in the absence of any information the prior distribution remains unchanged. The MaxEnt principle then represents a special case of the MinRel principle for discrete variables and uniform prior distribution.

In recent years, there has been much debate as to whether generalized entropies can fulfill SJ axioms, and if yes, how the permissible classes are classified (see, e.g., presse ; tsallis ; presse2 and citations therein). In their latest paper bagci , Oikinomou and Bagci (OB) focused on the particular case of Rényi’s entropy (RE) and argued that RE is not consistent with some of SJ axioms and hence it is ill-suited for inference purposes. This finding is, however, at odds with recently found one-parameter class of (entropic) functionals — so-called Uffink’s class, which is consistent with SJ axioms jizba19 , and which contains RE as a particular member. In addition, if the OB statement was true, then in some important cases, such as in the Rényi entropy-based signal processing and pattern recognition, there would be important new corrections or inconsistencies to some existing analyzes. OB result would be also detrimental in quantum information theory puchala ; Zyczkowski (e.g., RE for $q=2$ is related to purity). Here we show that the RE as they stand is certainly compatible with SJ axioms. Rather than appealing to Ref. jizba19 for a full-fledged proof of Uffink’s class, we will employ more straightforward approach. In particular, in this Comment we directly point out several issues in OB reasonings. We carefully go through OB arguments and correct respective problematic points. When all issues are properly accounted for, no violation of SJ is found. For the sake of simplicity, we focus on the discrete version of SJ axioms and change to continuous variables only when necessary.

In the following we will denote the RE of order $q$ as

[TABLE]

and the ensuing relative RE (or Rényi’s divergence of order $q$) 111Notice that the relative RE exists in two versions (both proposed by Rényi himself Renyi). In our reasonings here we stick to 1st Rényi’s version which is also employed in the OB paper., as

[TABLE]

Critical revision of the OB paper. —  Let us now go step-by-step through the key arguments presented by OB in bagci . Our discussion will be organized in the descendent order according to respective SJ axiomatic points:

1. Uniqueness axiom —

OB conclude that the first axiom is fulfilled only for $q\in(0,1)$, since only for such values is RE concave. The latter is certainly true, however concavity is only sufficient not necessary condition for uniqueness. A key observation in this context is that RE is a strictly Schur-concave function for arbitrary $q>0$ 222In general, when a given function $H$ is strictly Schur-concave this means that for two probability vectors satisfying the majorization relation $p\prec q$, (i.e., $\sum_{i=1}^{k}p_{(i)}\leq\sum_{i=1}^{k}q_{(i)}$ for $k\in\{1,\dots,n-1\}$, where $p_{(i)}$ and $q_{(i)}$ are the ordered distributions components in the descending order) one has the inequality $H(q)\geq H(p)$ where $H(p)=H(q)$ only if $p=q$. It should be stressed that a Schur-concave function need not be concave and concave function need not be Schur-concave. However, a symmetric concave function is always Schur-concave. Both concepts of concavity and Schur-concavity coincide for functions of one variable. which is also sufficient condition cf. puchala ; Zyczkowski . Let us assume that there exist two distinct distributions $P_{1}$ and $P_{2}$ (which are not permutation of each other) that maximize $H_{q}$ under given constraints. Let us take a convex combination $P_{\alpha}=\alpha P_{1}+(1-\alpha)P_{2}$. Indeed, $P_{\alpha}$ belongs to the probability simplex and also fulfills the constraints. Since $H_{q}$ is strictly Schur-concave, it fulfills the following inequality (see also roberts )

[TABLE]

Thus, the result must be unique otherwise we get contradiction with maximality assumption uffink . This fact will also be important in connection with subset independence axiom.

2. Invariance axiom —

For discrete case, the permutation invariance axiom means that the entropy should be symmetric function of probabilities, which is indeed the case for RE since RE is Schur-concave. Let us recall that every concave and symmetric function is Schur-concave. The opposite implication is not true, but all Schur-concave functions (including RE) are symmetric (under permutation of the arguments) Wayne .

For continuous variables, one should use (2). The latter is manifestly invariant under the change of coordinate system ${\boldsymbol{x}}\mapsto{\boldsymbol{y}}$. Indeed, if ${\boldsymbol{y}}={\boldsymbol{\varphi}}({\boldsymbol{x}})$ and ${\boldsymbol{\varphi}}$ is a bijective, differentiable function then the well known transformation rule for probability density functions feller states that $p_{Y}({\boldsymbol{y}})=p_{X}({\boldsymbol{x}})|\det(\partial{{\boldsymbol{\varphi}}}^{-1}/\partial{\boldsymbol{y}})|$. By setting $p({\boldsymbol{x}})\equiv p_{X}({\boldsymbol{x}})$ and $q({\boldsymbol{x}})\equiv q_{X}({\boldsymbol{x}})$, and plugging this to (2) wee see that the latter is invariant under the change ${\boldsymbol{x}}\mapsto{\boldsymbol{y}}$. One could even be more general and employ Radon–Nikodym theorem RDa . With this the Rényi’s divergence of order $q$ from $P$ to $Q$ can be rewritten as

[TABLE]

where $dP/dQ$ is the Radon–Nikodym derivative. In this formulation is $H_{q}(P||Q)$ manifestly coordinate-system independent.

3. Subset independence axiom —

Here OB argue that the RE does not fulfill the subset independence axiom. To support their claim they use the Livesey–Skilling criterion livesey : Any inference (entropic) functional is consistent with subset independence axiom if for $j\neq k\neq l$ the following identity holds

[TABLE]

By using (5), OB show that RE does not fulfill this criterion and therefore does not conform with the subset independence axiom. It is, however, not difficult to see that (5) does not cover all possible configurations and as it stands it is too restrictive. In fact, in shore has been shown that the entropic functional satisfying first three SJ axioms must be of the form

[TABLE]

where $f$ is an arbitrary increasing function and $g$ is an increasing concave function. Above entropic functionals are called sum-form entropies. The proof can be found in the Supplemental material of jizba19 . Note that the explicit form of a function $f$ in (6) does not influence the form of the distribution estimated by the MaxEnt principle. This can be easily seen by comparing two situations: a) $f(x)=ax$ (with $a>0$ being a constant), in which case the MaxEnt principle dictates that we should maximize $\sum_{i}g(p_{i})$ subject to constraints $\sum_{i}p_{i}=1$ and $\sum_{i}p_{i}E_{i}=E$. This gives

[TABLE]

and therefore

[TABLE]

b) $f\neq ax$, where the MaxEnt principle prescribes that we should maximize $f(\sum_{i}g(p_{i}))$ under the same constraints as above. In this case we have

[TABLE]

where $C_{f}\equiv f^{\prime}\left(\sum_{i}g(p_{i})\right)$. This leads to

[TABLE]

so the resulting MaxEnt distribution is indeed independent of $f$. This defines the equivalent classes of entropic functionals with equivalence $f(\sum_{i}g(p_{i}))\sim\sum_{i}g(p_{i})$.

Let us now go back to the criterion (5) and apply it to the class of entropic functionals (6). The difference of derivatives gives

[TABLE]

and successive derivative with respect to $p_{l}$ then yields

[TABLE]

The second term on the LHS vanishes for $l\neq j,k$. The first term is zero only when ${\partial C_{f}(P)}/{\partial p_{l}}=0$, which implies that $f^{\prime\prime}(x)=0$, or equivalently $f(x)=ax$. Consequently, we see that the Livesey–Skilling criterion employed by OB can support only the trace-class entropies, i.e., entropy functionals of the form $\sum_{i}g(p_{i})$. On the other hand, since the function $f$ does not change the resulting MaxEnt distribution, all sum-form entropies must be consistent with the subset independence axiom. The only quantities that are changed are the Lagrange parameters. The transform $\sum_{i}g(p_{i})\mapsto f\left(\sum_{i}g(p_{i})\right)$ can therefore be interpreted as a kind of gauge invariance in the MaxEnt principle.

Let us finally make two remarks regarding the aforementioned gauge invariance:

• •

Since the function $f$ can be arbitrary, it is irrelevant whether the entropic functional is additive or not. Actually, by application of appropriate function, one can impose the desired type of (generalized) additivity. For example, the RE is additive, while Tsallis entropy tsallis2 , $S_{q}=\ln_{q}\exp H_{q}$ is $q$-additive (here $\ln_{q}$ is the $q$-deformed logarithm borges ) and RE power jizba , $P_{q}=\exp H_{q}$ is multiplicative.

• •

From the sum-form (6) of entropy it is also clear that concavity cannot be necessary condition for uniqueness, since an increasing function of a concave function does not need to be concave. On the other hand, an increasing function of any Schur-concave function is yields again Schur-concave function schur ; Wayne .

4. System independence axiom —

OB also conclude that RE does not fulfill system independence axiom. Their result relies on the solution of functional equation

[TABLE]

where $p_{ij}$ is the joint distribution of the whole system and $u_{i}$ and $v_{j}$ are the marginal distributions of two disjoint subsystems. At this point, OB follow the approach of Pressé et al. presse and assume the form of the joint distribution as $p_{ij}=u_{i}v_{j}$. From this, they end up with the equation $xg^{\prime\prime\prime}(x)+g^{\prime\prime}(x)=0$ leading to $g(x)=-x\log x$ modulo multiplicative and additive constant, which corresponds to Shannon entropy. However, as already discussed in uffink ; jizba19 , the assumption on the structure of probability distribution goes well beyond the original idea of consistency axioms, since assume only certain structure of updating information, not the probability distribution itself. In Ref. jizba19 it was pointed out that this requirement can be ensured by assuming a stronger version of the system independence axiom, which can be formulated as follows: whenever two subsystems of a system are disjoint, we can treat the subsystems in terms of independent distributions. We shall note that this strong system independence axiom is fulfilled for many systems observed in nature, namely for systems which state space scales exponentially. These systems have typically short-range interactions. In this case, the strong independence axiom allows to bring Eq. (17) to the following form:

[TABLE]

On the other hand, this is apparently not the most general form of the relation between joint and marginal distribution. It is beyond the scope of this Comment to investigate the general form of this relation, but let us just look at the case when the ratio is also linear but with some prefactor. In this case, we have:

[TABLE]

for $a$ close to $1$. This leads to differential equation in the form $axg^{\prime\prime\prime}(x)+g^{\prime\prime}(x)=0$ which has the solution $g(x)\propto x^{2-1/a}$. By denoting $q=2-1/a$, i.e., $a=1/(2-q)$, we end up with entropic functional equivalent to RE. Eq. (19) corresponds to the composition rule of $q$-exponential distributions described in jizba19

[TABLE]

Reader can easily check that (20) satisfies Eq. (17). As shown in uffink (see also Supplemental Material in Ref. jizba19 ), RE conforms with the system independence axiom.

5. Maximality axiom —

strict Schur-concavity of RE automatically ensures that in the absence of any prior information, the uniform distribution must be the MaxEnt distribution. Indeed, $H_{q}(P)\geq H_{q}(Q)$ whenever $P\prec Q$ for any two $n$-dimensional distributions $P$ and $Q$, and hence

[TABLE]

for any $P$. This is a simple consequence of observation that

[TABLE]

Note that the lover limit in (21) is saturated only when $P$ is a permutation of the pure-state distribution $\{1,0,\ldots,0\}$ and the upper limit is saturated only for maximally-mixed-state (uniform) distribution $P=\{1/n,\ldots,1/n\}$. So RE has a strict global maximum at the uniform distribution. This completes the proof. Relations (21)-(22) also nicely bolster the usual interpretation of entropy — the larger is the entropy, the more uniform is the distribution.

The issue of escort constraints. —  Let us also briefly comment the use of the escort averages $\sum_{i}\rho_{q}(p_{i})E_{i}=E_{q}$ in MaxEnt prescription. It should be first stressed that the whole framework of SJ consistency axioms has been invented for the case of linear constraints $\sum_{i}p_{i}E_{i}=E$. Its extension to more generalized types of constraints as, for example, escort averages remains an open problem. Therefore, it is not possible to apply the original SJ criteria to that situation. One possible way how to overcome usage of escort means is to change the probability distribution to the escort distribution, so that the constraints become linear. In this case, one can also formulate the entropy in terms of escort distributions $p_{i}=\rho_{i}^{1/q}/\sum_{j}\rho_{j}^{1/q}$, which leads to maximization of functional equivalent to the Landsberg (or also Homogenous) entropy functional lutsko ; Landsberg

[TABLE]

When the whole framework is formulated in terms of escort probabilities with linear constraints, we can employ the SJ axioms but formulated in terms of escort distributions. According to jizba19 , Eq. (23) will then belong to the class of Uffink’s entropic functionals (i.e., class that is consistent with SJ axioms). This fact alone, however, does not ensure that the SJ axioms will also be valid in the original picture. Particularly, the problem might arise in connection with subset independence axioms, since the de Finneti–Kolmogorov theorem $p_{ij}=u_{i}v_{j|i}$ does not generally hold for corresponding escort-distribution counterparts jizba17 ; jizba17b .

Conclusions. —  In this Comment we have analyzed the recent claim of T. Oikinomou and B. Bagci bagci , that Rényi entropy is ill-suited for inference purposes because it is not consistent with the Shore–Johnson axioms. By carefully examining Oikinomou–Bagci arguments we have, however, noticed that there are several issues in their reasonings that need to be clarified. When the latter are properly accounted for we find that there is no contradiction with SJ desiderata. This conclusion should also be expected on more general ground. Namely, Rényi entropy is known to be a bona fide member of the so-called Uffink’s class of entropic functionals jizba19 ; uffink , which is the most general class of inference functionals satisfying SJ axioms jizba19 .

Import ingredient in our reasonings was the strict Schur-concavity of Rényi entropy. In fact, the language of majorization and (strict) Schur-concavity is very natural in the context of entropies since many processes in physics occur in the direction of the majorization arrow (because the passage of time tends to make things more uniform) and Schur-concave entropies grasp this behavior via their non-decreasing evolution.

Acknowledgements.

Acknowledgments. — P.J. and J.K. were supported by the Czech Science Foundation (GAČR), Grant 19-16066S. J.K. was also supported by the Austrian Science Foundation (FWF) under project I3073.