Tsallis entropy and generalized Shannon additivity

TL;DR

This paper examines the axiomatic foundations of Tsallis entropy, focusing on the generalized Shannon additivity axiom, and characterizes Tsallis entropy for most parameter values.

Contribution

It provides a simplified axiomatic characterization of Tsallis entropy based on Shannon additivity, excluding specific cases where $ ext{α}=1,2$.

Findings

Shannon additivity characterizes Tsallis entropy for most parameters.

The cases $ ext{α}=1,2$ require separate analysis.

The axiomatic approach clarifies the foundational differences from Shannon entropy.

Abstract

The Tsallis entropy given for a positive parameter $\alpha$ can be considered as a modification of the classical Shannon entropy. For the latter, corresponding to $\alpha=1$, there exist many axiomatic characterizations. One of them based on the well-known Khinchin-Shannon axioms has been simplified several times and adapted to Tsallis entropy, where the axiom of (generalized) Shannon additivity is playing a central role. The main aim of this paper is to discuss this axiom in the context of Tsallis entropy. We show that it is sufficient for characterizing Tsallis entropy with the exceptions of cases $\alpha=1,2$ discussed separately.

Equations

$$S=-k_B\sum _{i=1}^n{p}_i\ln p_i,$$

$$S^{\alpha }=k_B\frac{1-{\sum }_{i=1}^n{p}_i^{\alpha }}{\alpha -1}$$

$$H(p_1,p_2,\dots ,p_n)=\frac{1-{\sum }_{i=1}^n{p}_i^{\alpha }}{\alpha -1}.$$

$$H(p_1,p_2,\dots ,p_n)=-\sum _{i=1}^n{p}_i\ln p_i,$$

$$H\text{mbox}iscontinuouson{△}_n\text{mbox}foralln\in \mathbb{N},$$

$$H\left(\frac{1}{n},\frac{1}{n},\dots ,\frac{1}{n}\right)\ge H(p_1,p_2,\dots ,p_n)\text{mbox}forall(p_1,p_2,\dots ,p_n)\in {△}_n,$$

$$H(p_1,p_2,\dots ,p_n,0)=H(p_1,p_2,\dots ,p_n)\text{mbox}forall(p_1,p_2,\dots ,p_n)\in △,$$

$$H(p_{1,1},...,p_{1,m_1},p_{2,1},...,p_{2,m_2},...,p_{n,1},...,p_{n,m_n})=H(p_1,...,p_n)+\sum _{i=1}^n{p}_{i}H\left(\frac{p_{i,1}}{p_i},...,\frac{p_{i,m_i}}{p_i}\right)$$

$$\text{mbox}forall(p_{1,1},...,p_{1,m_1},p_{2,1},...,p_{2,m_2},...,p_{n,1},...,p_{n,m_n})\in △;n,m_1,\dots ,m_n\in \mathbb{N}$$

$$\text{mbox}and{p}_i=\sum _{j=1}^{m_i}{p}_{i,j};j=1,...,m_i.$$

$$H(p_{1,1},...,p_{1,m_1},p_{2,1},...,p_{2,m_2},...,p_{n,1},...,p_{n,m_n})=H(p_1,...,p_n)+\sum _{i=1}^n{p}_i^{\alpha }H\left(\frac{p_{i,1}}{p_i},...,\frac{p_{i,m_i}}{p_i}\right)$$

$$\text{mbox}forall(p_{1,1},...,p_{1,m_1},p_{2,1},...,p_{2,m_2},...,p_{n,1},...,p_{n,m_n})\in △;n,m_1,\dots ,m_n\in \mathbb{N}$$

$$\text{mbox}and{p}_i=\sum _{j=1}^{m_i}{p}_{i,j};j=1,...,m_i$$

$$H(p_1,...,p_{j-1},p_j,p_{j+1},p_{j+2},\dots ,p_n)=H(p_1,...,p_{j-1},p_j+p_{j+1},p_{j+2},\dots ,p_n)$$

$$+(p_j+p_{j+1}{)}^{\alpha }H\left(\frac{p_j}{p_j+p_{j+1}},\frac{p_{j+1}}{p_j+p_{j+1}}\right)$$

$$\text{mbox}forall(p_1,p_2,\dots ,p_n\in △);n\in \mathbb{N};j=1,2,\dots ,n-1.$$

$$H(p_1,p_2,\dots ,p_n)=H\left(\frac{1}{2},\frac{1}{2}\right)\frac{1-{\sum }_{i=1}^n{p}_i^{\alpha }}{1-2^{1-\alpha }}\text{mbox}forall(p_1,p_2,\dots ,p_n)\in △.$$

$$H(p_1,p_2,\dots ,p_n)=2H\left(\frac{1}{2},\frac{1}{2}\right)\left(1-\sum _{i=1}^n{p}_i^2\right)\text{mbox}forall(p_1,p_2,\dots ,p_n)\in △,$$

$$H(p_1,p_2,\dots ,p_n)=-H\left(\frac{1}{2},\frac{1}{2}\right)\left(\sum _{i=1}^n{p}_i{\log }_2{p}_i\right)\text{mbox}forall(p_1,p_2,\dots ,p_n)\in △,$$

$$(1-3\cdot 2^{-\alpha })H(p_1,p_2)+2^{-\alpha }H(p_2,p_1)=H\left(\frac{1}{2},\frac{1}{2}\right)(1-p_1^{\alpha }-p_2^{\alpha }),$$

$$H(p_1,p_2)=H(p_2,p_1)$$

$$H(p_1,p_2)+H(p_2,p_1)=4H\left(\frac{1}{2},\frac{1}{2}\right)(1-p_1^2-p_2^2).$$

$$H(1)=0.$$

$$H(1,0)=H(1)+1^{\alpha }H(1,0).$$

$$H\left(\frac{1}{2},\frac{1}{2}\right)+{\left(\frac{1}{2}\right)}^{\alpha }H(1,0)=H\left(\frac{1}{2},\frac{1}{2},0\right)=H(1,0)+1^{\alpha }H\left(\frac{1}{2},\frac{1}{2}\right).$$

$$H(p_1,p_2)+(p_1^{\alpha }+p_2^{\alpha })H\left(\frac{1}{2},\frac{1}{2}\right)=H\left(\frac{p_1}{2},\frac{p_1}{2},\frac{p_2}{2},\frac{p_2}{2}\right)=H\left(\frac{p_1}{2},\frac{1}{2},\frac{p_2}{2}\right)+2^{-\alpha }H(p_1,p_2)$$

$$H\left(\frac{p_1}{2},\frac{1}{2},\frac{p_2}{2}\right)+2^{-\alpha }H(p_2,p_1)=H\left(\frac{p_1}{2},\frac{p_2}{2},\frac{p_1}{2},\frac{p_2}{2}\right)=H\left(\frac{1}{2},\frac{1}{2}\right)+2^{1-\alpha }H(p_1,p_2).$$

$$H(p_1,p_2)+(p_1^{\alpha }+p_2^{\alpha })H\left(\frac{1}{2},\frac{1}{2}\right)=H\left(\frac{1}{2},\frac{1}{2}\right)+3\cdot 2^{-\alpha }H(p_1,p_2)-2^{-\alpha }H(p_2,p_1),$$

$$H(p_1,...,p_{j-1},p_j,p_{j+1},p_{j+2},\dots ,p_n)=H(p_1,...,p_{j-1},p_{j+1},p_j,p_{j+2},\dots ,p_n)$$

$$\text{mbox}forall(p_1,p_2,\dots ,p_n)\in △;n\in \mathbb{N};j=1,2,\dots ,n-1$$

$$H(p_1,p_2)=H\left(\frac{1}{2},\frac{1}{2}\right)\frac{1-p_1^{\alpha }-p_2^{\alpha }}{1-2^{1-\alpha }}\text{mbox}forall(p_1,p_2)\in {△}_2.$$

$$H(p_1,p_2)=2H\left(\frac{1}{2},\frac{1}{2}\right)(1-p_1^2-p_2^2)\text{mbox}forall(p_1,p_2)\in {△}_2,$$

$$(1-3\cdot 2^{-\alpha })H(p_2,p_1)=2^{\alpha }\left(2^{-\alpha }H\left(\frac{1}{2},\frac{1}{2}\right)(1-p_1^{\alpha }-p_1^{\alpha })-2^{-2\alpha }H\left(p_1,p_2\right)\right).$$

$$(1-3\cdot 2^{-\alpha })$$

$$=2^{\alpha }$$

$$(1-2^{2-\alpha })H\left(\frac{1}{2},\frac{1}{2}\right)(1-p_1^{\alpha }-p_2^{\alpha })$$

Full text

1 Introduction

Some history.

In 1988 Tsallis [16] generalized the Boltzmann-Gibbs entropy

[TABLE]

describing classical thermodynamical ensembles with microstates of probabilities $p_{i}$, by the entropy

[TABLE]

for $0<\alpha\neq 1$ in the sense that $\lim_{\alpha\to 1}S^{\alpha}=S$. Here $k_{B}$ is the Boltzmann constant (which as only a multiplicative constant will not be considered in the following). Many physicists argue that this was a breakthrough in thermodynamics since the extension allows better describing systems out of equilibrium and systems with strong correlations between microstates, but there is also critizism on the application of Tsallis’ concept (compare [7, 15]). In information theory pioneered by Shannon, the Boltzmann-Gibbs entropy is one of the central concept. We follow the usual practice to call it Shannon entropy. Also note that Tsallis’ entropy concept coincides up to a constant with the Havrda-Charvát entropy [8] given in 1967 in an information theoretical context.

There have been given many axiomatic characterications of Tsallis’ entropy starting from such of the classical Shannon entropy (see below). One important axiom called (generalized) Shannon additivity is extensively discussed and shown to be sufficient in some sense in this paper.

Tsallis entropy.

In the following, let $\triangle_{n}=\{(p_{1},p_{2},\ldots,p_{n})\in({\mathbb{R}}^{+})^{n},\sum_{i=1}^{n}p_{i}=1\}$ for $n\in{\mathbb{N}}$ be the set of all $n$-dimensional stochastic vectors and $\triangle=\bigcup_{n\in{\mathbb{N}}}\triangle_{n}$ be the set of all stochastic vectors, where ${\mathbb{N}}=\{1,2,3,\ldots\}$ and ${\mathbb{R}}^{+}$ are the sets of natural numbers and of nonnegative real numbers, respectively. Given $\alpha>0$ with $\alpha\neq 1$, the Tsallis entropy of a stochastic vector $(p_{1},p_{2},\ldots,p_{n})$ of some dimension $n$ is defined by

[TABLE]

In the case $\alpha=1$, the value $H\left(p_{1},\ldots,p_{n}\right)$ is not defined, but the limit of it as $\alpha$ approaches to $1$ is

[TABLE]

which provides the classical Shannon entropy. In so far Tsallis entropy can be considered as a generalization of the Shannon entropy and so it is not surprising that various axiomatic characterizations of the latter one have been tried to generalize to the Tsallis entropy.

Axiomatic characterizations.

One line of characterizations mainly followed by Suyari [14] and discussed in this paper has its origin in the Shannon-Khinchin axioms of Shannon entropy (see [13] and [10]). Note that other characterizations of Tsallis entropy are due to dos Santos [12], Abe [1] and Furuichi [6]. For some general discussion of axiomatization of entropies see [2].

A map $H:\triangle\rightarrow{\mathbb{R}}^{+}$ is the Shannon entropy up to a multiplicative positive constant if it satisfies the following axioms:

[TABLE]

Axiom (1) called Shannon additivity is playing a key role in the characterization of the Shannon entropy and an interesting result given by Suyari [14] says that its generalization

[TABLE]

for $\alpha\neq 1$ provides the Tsallis entropy for this $\alpha$.

More precisely, if $H:\triangle\rightarrow{\mathbb{R}}^{+}$ satisfies (S1), (S2), (S3) and (1), then $c(\alpha)H$ is the Tsallis entropy for some positive constant $c(\alpha)$. The full result of Suyari, which was slightly corrected by Ilić et al. [9] includes a characterization of the map $\alpha\rightarrow c(\alpha)$ under the assumption that $H$ also depends continuously on $\alpha\in{\mathbb{R}}^{+}\setminus\{0\}$. We do not discuss this characterization, but we note here that the results below also provide an immediate simplification of the whole result of Suyari and Ilić et al.

The main result.

In this paper, we study the role of generalized Shannon additivity in characterizing Tsallis entropy, where for $\alpha\in{\mathbb{R}}^{+}\setminus\{0\}$ and $H:\triangle\rightarrow{\mathbb{R}}$ we also consider the slightly relaxed property that

[TABLE]

It turns out that this property basically is enough for characterizing the Tsallis entropy for $\alpha\in{\mathbb{R}}^{+}\setminus\{0,1,2\}$ and with a further weak assumption in the cases $\alpha=1,2$. As already mentioned, the statement (iii) for $\alpha=1$ is an immediate consequence of a characterization of Shannon entropy by Diderrich [4] simplifying an axiomatization given by Faddeev [5] (see below).

Theorem 1**.**

Let $H:\triangle\rightarrow{\mathbb{R}}$ be given with (1), or a bit weaker (1), for $\alpha\in{\mathbb{R}}^{+}\setminus\{0\}$. Then the following holds:

1. (i)

If $\alpha\neq 1,2$, then

[TABLE] 2. (ii)

If $\alpha=2$, then the following statements are equivalent:

1. (a)

It holds

[TABLE] 2. (b)

$H$* is bounded on $\triangle_{2}$,* 3. (c)

$H$* is continuous on $\triangle_{2}$,* 4. (d)

$H$* is symmetric on $\triangle_{2}$,* 5. (e)

$H$* does not change the signum on $\triangle_{2}$.* 3. (iii)

If $\alpha=1$, then the following statements are equivalent:

1. (a)

It holds

[TABLE] 2. (b)

$H$* is bounded on $\triangle_{2}$.*

Note that statement (iii) is given here only for reasons of completeness. It follows from a result of Diderrich [4].

The paper is organized as follows. Section 2 is devoted to the proof of the main result. It will turn out that most of the substantial work is related to stochastic vectors contained in $\triangle_{2}\cup\triangle_{3}$ and that the generalized Shannon additivity performs as a bridge to stochastic vectors longer than $2$ or $3$. Section 3 completes the discussion. In particular, the Tsallis entropy for $\alpha=1,2$ on rational vectors is discussed and an open problem is formulated.

2 Proof of the main result

We start with investigating the relationship of $H(p_{1},p_{2})$ and $H(p_{2},p_{1})$ for $(p_{1},p_{2})\in\triangle_{2}$.

Lemma 2**.**

Let $\alpha\in{\mathbb{R}}^{+}\setminus\{0\}$ and $H:\triangle\rightarrow{\mathbb{R}}$ satisfy (1). Then for all $(p_{1},p_{2})\in\triangle_{2}$ it follows

[TABLE]

in particular for $\alpha=1$

[TABLE]

and for $\alpha=2$

[TABLE]

Moreover it holds

[TABLE]

Proof.

First of all, note that (5) is an immediate consequence of (1) implying

[TABLE]

Further, two different applications of (1) to $H\left(\frac{1}{2},\frac{1}{2},0\right)$ provide

[TABLE]

Therefore $H(1,0)=0$, and since similarly one gets $H(0,1)=0$, in the following we can assume that $p_{1},p_{2}\neq 0$.

Applying (1) three times, one obtains

[TABLE]

and in the same way

[TABLE]

Transforming (7) to the term $H\left(\frac{p_{1}}{2},\frac{1}{2},\frac{p_{2}}{2}\right)$ and then substituting this term in (6), provides

[TABLE]

which is equal to (2). Statements (3) and (4) follow immediately from equation (2). ∎

In the case $\alpha=1$ condition (1) implies that the order of components of a stochastic vector does not make a difference for $H$:

Lemma 3**.**

Let $H:\triangle\rightarrow{\mathbb{R}}$ satisfy (1) for $\alpha=1$. Then $H$ is permutation-invariant, meaning that $H(p_{1},p_{2}\ldots,p_{n})=H(p_{\pi(1)},p_{\pi(2)}\ldots,p_{\pi(n)})$ for each $(p_{1},p_{2},\ldots,p_{n})\in\triangle;\,n\in{\mathbb{N}}$ and each permutation $\pi$ of $\{1,2,\ldots,n\}$.

Proof.

It suffices to show that

[TABLE]

For $n<3$ this has been shown in Lemma 2 (see (3)), for $n\geq 3$ it follows directly from (1) and from Lemma 2. ∎

The following lemma provides the substantial part of the proof of Theorem 1.

Lemma 4**.**

For $H:\triangle\rightarrow{\mathbb{R}}$ satisfying (1) with $\alpha\in{\mathbb{R}}^{+}\setminus\{0,1\}$, the following holds:

1. (i)

If $\alpha\neq 2$, then

[TABLE] 2. (ii)

If $\alpha=2$, then the following statements are equivalent:

1. (a)

It holds

[TABLE] 2. (b)

$H$* is symmetric on $\triangle_{2}$, meaning that $H(p_{1},p_{2})=H(p_{2},p_{1})$ for all $(p_{1},p_{2})\in\triangle_{2}$,* 3. (c)

$H$* is continuous on $\triangle_{2}$,* 4. (d)

$H$* is bounded on $\triangle_{2}$,* 5. (e)

$H$* is nonnegative or nonpositive on $\triangle_{2}$.*

Proof.

We first show (i). Let $\alpha\neq 2$ and $(p_{1},p_{2})\in\triangle_{2}$. Changing the role of $p_{1}$ and $p_{2}$ in (2), by Lemma 2 one obtains

[TABLE]

Moreover, one easily sees that (2) transforms to

[TABLE]

(8) and (9) provide

[TABLE]

Since $1-3\cdot 2^{1-\alpha}+2^{3-2\alpha}=(1-2^{2-\alpha})(1-2^{1-\alpha})$, it follows

[TABLE]

In order to show (ii), let $\alpha=2$ and define maps $f:[\frac{1}{2},1]\rightarrow[\frac{1}{2},1]$ and $D:[\frac{1}{2},1]\rightarrow[0,\infty[$ by

[TABLE]

and

[TABLE]

for $p\in[\frac{1}{2},1]$.

By (4) in Lemma 2, (a) is equivalent both to (b) and to $D(p)=0$ for all $p\in[\frac{1}{2},1]$. (c) implies (d) by compactness of $\triangle_{2}$ and validity of the implications (a) $\Rightarrow$ (c) and (a) $\Rightarrow$ (e) is obvious.

From

[TABLE]

for $p\in\,\left[\frac{1}{2},1\right]$ one obtains

[TABLE]

and by induction

[TABLE]

with $f^{\circ n}(p)=\overbrace{f(f(\ldots(f(}^{n\mbox{\footnotesize\ times}}p))\ldots)).$

For $p\in\,\left]\frac{2}{3},1\right[$ it holds $f(p)=2-\frac{1}{p}$, hence $f$ maps the interval $\left]\frac{2}{3},1\right[$ onto the interval $\left]\frac{1}{2},1\right[$. Since $p-f(p)=\frac{(p-1)^{2}}{p}>0$ for all $p\in\,\left]\frac{2}{3},1\right[$, the following holds:

[TABLE]

Moreover, applying (10) to $p=\frac{1}{2}$ yields $0=D(\frac{1}{2})=\frac{D(1)}{4}$, hence

[TABLE]

Assuming (d), by use of (11), (12) and (13) one obtains $D(p)=0$ for all $p\in\,\left[\frac{1}{2},1\right]$, hence (a). If (e) is valid, then by (4) in Lemma 2

[TABLE]

for all $r\in[\frac{1}{2},1]$, providing (d). By the already shown, (a), (b), (c), (d), (e) are equivalent . ∎

We are able now to complete the proof of Theorem 1. Assuming (1), we first show (1) for $\alpha\neq 1,2$, and for $H$ bounded and $\alpha=2$. This provides statement (i) and, together with Lemma 4 (ii), statement (ii) of Theorem 1.

Statement (1) is valid for all $(p_{1},p_{2},\ldots,p_{n})\in\triangle_{1}\cup\triangle_{2}$ by Lemma 4. In order to prove it for $n>2$, we use induction. Assuming validity of (1) for all $(p_{1},p_{2},\ldots,p_{n})\in\triangle$ with $n=k$, where $k\in{\mathbb{N}}\setminus\{1\}$, let $(p_{1},p_{2},\ldots,p_{k},p_{k+1})\in\triangle$. Choose some $j\in\{1,2,\ldots,k\}$ with $p_{j}+p_{j+1}>0$. Then by (1) and Lemma 4 we have

[TABLE]

So (1) holds for all $(p_{1},p_{2},\ldots,p_{n})\in\triangle$ with $n=k+1$.

In order to see (iii), recall a result of Diderrich [4] stating that $H:\triangle\rightarrow{\mathbb{R}}$ is a multiple of the Shannon entropy if $H$ is bounded and permutation-invariant on $\triangle_{2}$ and satisfies

[TABLE]

which is weaker than (1) with $\alpha=1$. Since under (1) $H$ is permutation-invariant by Lemma 3, Diderrichs axiom are satisfied, and we are done.

3 Further discussion

Our discussion suggests that the case $\alpha=2$ is more complicated than the general one. In order to get some further insights, particularly in the case $\alpha=2$, let us consider consider only rational stochastic vectors. So in the following let $\triangle^{\mathbb{Q}}=\bigcup_{n\in{\mathbb{N}}}\triangle_{n}^{\mathbb{Q}}$ with $\triangle_{n}^{\mathbb{Q}}=\triangle_{n}\cap{\mathbb{Q}}^{n}$ for $n\in{\mathbb{N}}$ and ${\mathbb{Q}}$ being the rationals. The following proposition states that for $\alpha\neq 1$ the ‘rational’ generalized Shannon additivity principally provides the Tsallis entropy on the rationals, which particularly provides a proof of the implication (c) $\Rightarrow$ (a) in Theorem 1 (ii).

Proposition 5**.**

Let $H:\triangle^{\mathbb{Q}}\rightarrow{\mathbb{R}}$ be given with (1) for $\triangle^{\mathbb{Q}}$ instead of $\triangle$ and $\alpha\in{\mathbb{R}}^{+}\setminus\{0,1\}$. Then it holds

[TABLE]

Proof.

For the vectors $\left(\frac{1}{m},\ldots,\frac{1}{m}\right),\left(\frac{1}{n},\ldots,\frac{1}{n}\right)\in\triangle$ with $m,n\in{\mathbb{N}}$, we get from axiom (1)

[TABLE]

implying

[TABLE]

Now consider any rational vector $(p_{1},p_{2},\ldots,p_{n})\in\triangle^{\mathbb{Q}}$ with $p_{1}=\frac{a_{1}}{b},p_{2}=\frac{a_{2}}{b},...,p_{n}=\frac{a_{n}}{b}$ for $b,a_{1},\ldots,a_{n}\in{\mathbb{N}}$ satisfying $\sum\limits_{i=1}^{n}a_{i}=b$. With (1) we get

[TABLE]

Using (15), we obtain

[TABLE]

Let us finally compare (ii) and (iii) in Theorem 1 and ask for the role of (c), (d) and (e) of (ii) in (iii). Symmetry is already given by Lemma 3 when only (1) is satisfied, (1) and nonnegativity are not sufficient for characterizing Shannon entropy, as shown in [3]. By our knowledge, there is no proof that (1) and continuity are enough, but (1) and analyticity is working. Showing the latter, in [11] an argumentation reducing everything to the rationals as above has been used.

We want to resume with the open problem whether the further assumptions for $\alpha=2$ in Theorem 1 are necessary.

Problem**.**

Is (1) in Theorem 1 also valid for $\alpha=2$?