Rescuing the MaxEnt treatment for $q$-generalized entropies

TL;DR

This paper demonstrates that the MaxEnt variational approach can be effectively applied to Renyi's and Tsallis' entropies when formulated correctly, countering recent claims of its inadequacy.

Contribution

It provides a constructive reformulation of the MaxEnt problem for q-generalized entropies, showing its validity with an orthodox functional approach.

Findings

MaxEnt can be applied to q-entropies with proper formulation

Counteracts recent claims of MaxEnt's inadequacy for q-entropies

Provides a constructive method for variational problems in generalized entropies

Abstract

It has been recently argued, via very clever arguments, that the MaxEnt variational problem would not adequately work for Renyi's and Tsallis' entropies. We constructively show here that this is not so if one formulates the associated variational problem in a more orthodox functional fashion.

Equations

$$P_i=f(a+\beta {ϵ}_i)/Z,$$

$$Z=\sum _{i}f(a+\beta {ϵ}_i).$$

$$F_S(P)=-\sum _{i=1}^n{P}_i^q{\ln }_q(P_i)+{\lambda }_1\left(\sum _{i=1}^n{P}_i{U}_i-<U>\right)+{\lambda }_2\left(\sum _{i=1}^n{P}_i-1\right)$$

$$F_S(P+h)-F_S(P)=-\sum _{i=1}^n(P_i+h_i{)}^q{\ln }_q(P_i+h_i)+{\lambda }_1\left[\sum _{i=1}^n(P_i+h_i)U_i-<U>\right]+$$

$${\lambda }_2\left[\sum _{i=1}^n(P_i+h_i)-1\right]+\sum _{i=1}^n{P}_i^q{\ln }_q(P_i)-{\lambda }_1\left(\sum _{i=1}^n{P}_i{U}_i-<U>\right)-$$

$${\lambda }_2\left(\sum _{i=1}^n{P}_i-1\right)$$

$$F_S(P+h)-F_S(P)=\sum _{i=1}^n\left[\left(\frac{q}{1-q}\right)P_i^{q-1}+{\lambda }_1{U}_i+{\lambda }_2\right]h_i$$

$$\sum _{i=1}^{n}q{P}_i^{q-2}\frac{h_i^2}{2}+O(h^3),$$

$$\left(\frac{q}{1-q}\right)P_i^{q-1}+{\lambda }_1{U}_i+{\lambda }_2=0,$$

$$-\sum _{i=1}^{n}q{P}_i^{q-2}{h}_i^2\le C||h|{|}^2$$

$${\lambda }_1=\beta q{Z}^{1-q}$$

$${\lambda }_2=\frac{q}{q-1}{Z}^{1-q}$$

$$P_i=\frac{[1+\beta (1-q)U_i{]}^{\frac{1}{q-1}}}{Z}$$

$$Z=\sum _{i=1}^n[1+\beta (1-q)U_i{]}^{\frac{1}{q-1}}.$$

$$S_R=\frac{1}{1-\alpha }\ln \left(\sum _{i=1}^n{P}_i^{\alpha }\right),$$

$$F_{S_R}(P)=\frac{1}{1-\alpha }\ln \left(\sum _{i=1}^n{P}_i^{\alpha }\right)+{\lambda }_1\left(\sum _{i=1}^n{P}_i{U}_i-<U>\right)+{\lambda }_2\left(\sum _{i=1}^n{P}_i-1\right),$$

$$F_{S_R}(P+h)=\frac{1}{1-\alpha }\ln \left[\sum _{i=1}^n(P_i+h_i{)}^{\alpha }\right]+{\lambda }_1\left[\sum _{i=1}^n(P_i+h_i)U_i-<U>\right]+$$

$${\lambda }_2\left[\sum _{i=1}^n(P_i+h_i)-1\right],$$

$$F_{S_R}(P+h)-F_{S_R}(P)=\frac{1}{1-\alpha }\ln \left[\sum _{i=1}^n(P_i+h_i{)}^{\alpha }\right]-\frac{1}{1-\alpha }\ln \left(\sum _{i=1}^n{P}_i^{\alpha }\right)+$$

$${\lambda }_1\sum _{i=1}^n{h}_i{U}_i+{\lambda }_2\sum _{i=1}^n{h}_i.$$

$$F_{S_R}(P+h)-F_{S_R}(P)=\frac{1}{1-\alpha }\ln \left\{\sum _{i=1}^n\left[P_i^{\alpha }+\alpha h_i{P}_i^{\alpha -1}+\frac{\alpha (\alpha -1)}{2}{h}_i^2{P}_i^{\alpha -2}\right]\right\}-$$

$$\frac{1}{1-\alpha }\ln \left(\sum _{i=1}^n{P}_i^{\alpha }\right)+{\lambda }_1\sum _{i=1}^n{h}_i{U}_i+{\lambda }_2\sum _{i=1}^n{h}_i.$$

$$F_{S_R}(P+h)-F_{S_R}(P)=\frac{1}{1-\alpha }\ln \left\{1+\frac{\sum _{i=1}^n\left[\alpha h_i{P}_i^{\alpha -1}+\frac{\alpha (\alpha -1)}{2}{h}_i^2{P}_i^{\alpha -2}\right]}{\sum _{i=1}^n{P}_i^{\alpha }}\right\}+$$

$${\lambda }_1\sum _{i=1}^n{h}_i{U}_i+{\lambda }_2\sum _{i=1}^n{h}_i,$$

$$F_{S_R}(P+h)-F_{S_R}(P)=\frac{1}{1-\alpha }\frac{\sum _{i=1}^n\left[\alpha h_i{P}_i^{\alpha -1}+\frac{\alpha (\alpha -1)}{2}{h}_i^2{P}_i^{\alpha -2}\right]}{\sum _{i=1}^n{P}_i^{\alpha }}+$$

$$-\frac{1}{2(1-\alpha )}{\left(\frac{\sum _{i=1}^n\alpha h_i{P}_i^{\alpha -1}}{\sum _{i=1}{P}_i^{\alpha }}\right)}^2+{\lambda }_1\sum _{i=1}^n{h}_i{U}_i+{\lambda }_2\sum _{i=1}^n{h}_i.$$

$$\frac{\alpha }{1-\alpha }\frac{P_i^{\alpha -1}}{\sum _{i=1}^n{P}_i^{\alpha }}+{\lambda }_1{U}_i+{\lambda }_2=0.$$

$$\frac{\alpha }{1-\alpha }\frac{P_i^{\alpha }}{\sum _{i=1}^n{P}_i^{\alpha }}+{\lambda }_1{P}_i{U}_i+{\lambda }_2{P}_i=0.$$

$$\frac{\alpha }{1-\alpha }+{\lambda }_1<U>+{\lambda }_2=0.$$

$${\lambda }_2=\frac{\alpha }{\alpha -1}-{\lambda }_1<U>,$$

Full text

Keywords: Tsallis entropy, Renyi entropy, MaxEnt

1 Introduction

Renyi’s and Tsallis’ information measures ($S_{R}$ and $S_{T}$), respectively) are generalizations of the Shannon one, quantifying a system’s diversity, uncertainty, or randomness. The degree of generalization is parameterized by a real parameter. Both are very important quantities in variegated areas of scientific research. One can mention, for instance, ecology, quantum information, the Heisenberg XY spin chain model, theoretical computer science, diffusion processes, several biological processes, high energy physics, etc. As a small sample, see for example, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22].

Information Theory (IT) yields a quite powerful inference methodology, abbreviated as MaxEnt [23], that is able to describe general properties of arbitrary systems, in most areas the of scientific endeavor, on the basis of scarce information. MaxEnt yields the least-biased description that can be gotten according to specific data-amounts, in any possible circumstances [23]. In statistical mechanics (SM), Jaynes pioneered the use of MaxEnt so as to both i) reformulate SM and ii) generalize its foundations [23]. Here we focus on the MaxEnt technique as applied to Renyi-Tsallis information measures.

A very interesting result was recently reported in [24]. Let $a$ and $\beta$ stand for the two Lagrange multipliers appearing in a canonical ensemble MaxEnt treatment. Then, one confidently expects MaxEnt to yield, for the canonical probability distribution (PD) $P$ ($Z$ being the partition function, $f$ an appropriate function, and $\epsilon$ the level energy),

[TABLE]

[TABLE]

It is claimed in [24], by recourse to an ingenious procedure, that for this to be possible $f$ must be endowed with certain properties. These properties, it is shown in [24], are NOT possessed by the PDs that maximize either Tsallis’ or Renyi’s entropies. This constitutes an intriguing result that deserves further consideration.

We show here by construction, appealing to conventional techniques of Functional Analysis, not used in [24], that the functional forms (1.1), (1.2) are indeed recovered à la Tsallis (or Renyi), thus surmounting the formidable obstacle posed by [24] to the use of MaxEnt for these two entropies. We are not claiming that the arguments of [24] are wrong, but that they are Calculus-based, while the proper MaxEnt scenario requires functional Analysis. The central notion required here is that of increment $h$ of a functional. The general theory of Variational Calculus is concerned with Banach Space (BS) [25], examples of which are Hilbert’s space and classical phase space. The MaxEnt technique in Banach space demands a first variation that has to vanish and a second one that ascertains the characteristics of the ensuing extremum. The main need is then to evaluate the increment $h$ of a functional $F$ at the point $x$ of the operative BS. One must remember that functional calculus is not identical to ordinary calculus (involving ordinary functions), particularly when one is looking for extremes [26, 25].

2 Tsallis MaxEnt Treatment

The concomitant Tsallis’ procedure has been developed in [26]. The Tsallis information functional is [11]

[TABLE]

For the increment one has

[TABLE]

[TABLE]

[TABLE]

Eq. (2.2) can be rewritten as

[TABLE]

[TABLE]

Eq. (2.3) leads to

[TABLE]

[TABLE]

Eq. (2.4) is the Euler-Lagrange one while (2.5) gives bounds originating from the second variation. Thus, (2.4) entails

[TABLE]

[TABLE]

The following two basic relations for our task follow now, namely,

[TABLE]

[TABLE]

These two relations should not exist according to [24]. That we can construct them follows from Functional Analysis, that was not appealed to in that reference.

3 Renyi’s MaxEnt Treatment

Renyi’s $S_{R}$ is defined as [9]:

[TABLE]

and the accompanying (canonical ensemble) MaxEnt probability distribution $P$ arises from the maximization of the functional $F_{S_{R}}(P)$ [where $U$ denotes the energy and $<U>$ its mean value]

[TABLE]

Following standard Functional Analysis procedures (which are not employed in Ref- [24], we consider the functional-$h$ increment [26, 25]

[TABLE]

[TABLE]

so that

[TABLE]

[TABLE]

We now tackle $h^{2}$ contributions so as to assess second variations of $F_{S_{R}}$ [26, 25]

[TABLE]

[TABLE]

or, equivalently,

[TABLE]

[TABLE]

so that one finally arrives at

[TABLE]

[TABLE]

Summing up, we have for the first variation

[TABLE]

Multiplying (3.8) by $P_{i}$ we find

[TABLE]

Integrating now we are led to

[TABLE]

This is an important result, showing that $\lambda_{1}$ and $\lambda_{2}$ are not independent Lagrange multipliers. We are authorized to write

[TABLE]

and replacing this value of $\lambda_{2}$ in (3.8) we get

[TABLE]

whose solution is given by

[TABLE]

[TABLE]

[TABLE]

Ref. [24] showed, via an alternative, calculus-based procedure, that Eqs. (3.14) and (3.15) are impossible to obtain in this form for Renyi’s entropy. We are able here to contradict such assertion because we employed Functional Analysis and not just ordinary calculus. Finally, note that Eqs. (8) and (11) of [24] define both the exponential and the logarithm functions. Thus, their co-implication is obvious.

4 Conclusions

The MaxEnt variational treatment in the canonical ensemble is believed to yield, for the probability distribution (PD) $P$ that maximizes the entropy the forms

[TABLE]

[TABLE]

where $a$ and $\beta$ are Lagrange multipliers, $Z$ is the partition function, $f$ is an appropriate function, and $\epsilon$ is a level-energy.

It is claimed in [24], by recourse to an admirable, Calculus-based procedure, that for this to be possible $f$ must be endowed with special properties. These properties, one finds in [24], are NOT possessed by the PDs that maximize either Tsallis’ or Renyi’s entropies.

We have shown here (by construction), appealing to conventional techniques of Functional Analysis, not used in [24], that the functional forms (1.1), (1.2) are indeed encountered in dealing with Tsallis’ (or Renyi’s) entropies. This removes the rather formidable obstacle posed by [24] to the use of MaxEnt for these two entropies. Note that we are not claiming that the arguments of [24] are wrong, but that they are Calculus-based, while the proper MaxEnt scenario requires functional Analysis.