The Generalized Boltzmann Distribution is the Only Distribution in Which the Gibbs-Shannon Entropy Equals the Thermodynamic Entropy

TL;DR

This paper proves that the generalized Boltzmann distribution uniquely ensures the equality of Gibbs-Shannon entropy and thermodynamic entropy, highlighting that this equality only occurs at thermodynamic equilibrium.

Contribution

It establishes a unique characterization of the generalized Boltzmann distribution based on entropy equality, clarifying the conditions for entropy equivalence.

Findings

Gibbs-Shannon entropy equals thermodynamic entropy only at equilibrium.

The generalized Boltzmann distribution is uniquely characterized by this entropy equality.

Entropy equality does not hold for non-equilibrium distributions.

Abstract

We show that the generalized Boltzmann distribution is the only distribution for which the Gibbs-Shannon entropy equals the thermodynamic entropy. This result means that the thermodynamic entropy and the Gibbs-Shannon entropy are not generally equal, but rather than the equality holds only in the special case where a system is in equilibrium with a reservoir.

Equations

$$p_k=\{\begin{array}{ll}\frac{1}{{\Omega }_E}& E_k=E\\ 0& \text{otherwise}\end{array}$$

$$p_i=\sum _j\frac{1}{{\Omega }_E}=\frac{{\Omega }_{\text{r}(E-E_i)}}{{\Omega }_E}\propto {\Omega }_{\text{r}(E-E_i)}$$

$${\Omega }_{\text{r}(E-E_i)}=\exp \left[S_{r(E-E_i)}/k_B\right]$$

$$S_{r(E-E_i)}\simeq S_{r(E)}-\left(\frac{\partial S_r}{\partial E_r}\right)E_i\simeq S_r(E)-\frac{E_i}{T}$$

$$\mathcal{L}=-\sum _i{p}_i\log p_i-\beta \left(\sum _i{p}_i{E}_i-E_0\right)+\alpha \left(\sum _i{p}_i-1\right)$$

$$0=\frac{\delta \mathcal{L}}{\delta p_i}=-1-\log p_i-\beta E_i+\alpha \Rightarrow p_i=C\cdot e^{-\beta E_i}$$

$$U=⟨E⟩=\sum _i{p}_i{E}_i=\frac{{\sum }_i{E}_i{e}^{-\beta E_i}}{{\sum }_i{e}^{-\beta E_i}}=-\frac{\partial \log Z}{\partial \beta }$$

$$S=\frac{U-F}{T}=-k_B\sum _i{p}_i\log p_i=k_{B}H$$

$$\left.\begin{array}[]{c}\left(\begin{array}[]{c}\text{Thermodynamic}\\ \text{first law}\end{array}\right)\\ +\\ \left(\begin{array}[]{c}\text{The Boltzmann}\\ \text{distribution}\end{array}\right)\end{array}\right\}\Rightarrow\left(\begin{array}[]{c}\text{Thermodynamic entropy}\\ \text{equals}\\ \text{Gibbs-Shannon entropy}\end{array}\right)$$

$$\left.\begin{array}[]{c}\left(\begin{array}[]{c}\text{Thermodynamic}\\ \text{first law}\end{array}\right)\\ +\\ \left(\begin{array}[]{c}\text{Thermodynamic entropy}\\ \text{equals}\\ \text{Gibbs-Shannon entropy}\end{array}\right)\end{array}\right\}\Rightarrow\left(\begin{array}[]{c}\text{The Boltzmann}\\ \text{distribution}\end{array}\right)$$

$$dU=TdS+\sum _{\eta =1}^n{X}_{\eta }d{\chi }_{\eta }+\sum _{\eta =1}^m{Y}_{\eta }d{y}_{\eta }$$

$$Pr\left(\vec{\omega }\right)\propto \exp \left[\sum _{\eta =1}^n\frac{X_{\eta }{x}_{\eta }^{\left(\vec{\omega }\right)}}{k_{B}T}-\frac{E^{\left(\vec{\omega }\right)}}{k_{B}T}\right]$$

$$\mathrm{Pr}\left(\vec{\omega }\right)\propto f\left(E^{\left(\vec{\omega }\right)},x_1^{\left(\vec{\omega }\right)},\dots ,x_n^{\left(\vec{\omega }\right)};T,X_1,\dots ,y_1,\dots \right)$$

$$Z=\sum _{\vec{\omega }}{f}_{\vec{\omega }}$$

$$\begin{gathered} S=-k_B\sum _{\vec{\omega }}\frac{f_{\vec{\omega }}}{Z}\log \left(\frac{f_{\vec{\omega }}}{Z}\right) \\ =-k_B\left\{\frac{{\sum }_{\vec{\omega }}{f}_{\vec{\omega }}\log f_{\vec{\omega }}}{Z}-\log Z\right\} \end{gathered}$$

$$dS={\left(\frac{\partial S}{\partial T}\right)}_{X_1,\dots ,X_n,y_1,\dots ,y_m}dT$$

$$\begin{gathered} {\left(\frac{\partial S}{\partial T}\right)}_{X_1,\dots ,X_n,y_1,\dots ,y_m} \\ =-k_B\left\{\sum _{\vec{\omega }}\frac{Z\cdot \frac{\partial f_{\vec{\omega }}}{\partial T}\cdot \log f_{\vec{\omega }}-\frac{\partial Z}{\partial T}\cdot f_{\vec{\omega }}\log f_{\vec{\omega }}}{Z^2}\right\} \\ =-k_B\sum _{\vec{\omega }}\frac{\partial }{\partial T}\left(\frac{f_{\vec{\omega }}}{Z}\right)\cdot \log \left(f_{\vec{\omega }}\right) \end{gathered}$$

$$⟨E⟩=\sum _{\vec{\omega }}\frac{f_{\vec{\omega }}}{Z}\cdot E^{\left(\vec{\omega }\right)}$$

$$\begin{gathered} d⟨E⟩={\left(\frac{\partial ⟨E⟩}{\partial T}\right)}_{X_1,\dots ,X_n,y_1,\dots ,y_m}dT \\ =\sum _{\vec{\omega }}\frac{\partial }{\partial T}\left(\frac{f_{\vec{\omega }}}{Z}\right)\cdot E^{\left(\vec{\omega }\right)}\cdot dT \end{gathered}$$

$$\begin{gathered} d⟨x_{\eta }⟩={\left(\frac{\partial ⟨x_{\eta }⟩}{\partial T}\right)}_{X_1,\dots ,X_n,y_1,\dots ,y_m}dT \\ =\sum _{\vec{\omega }}\frac{\partial }{\partial T}\left(\frac{f_{\vec{\omega }}}{Z}\right)\cdot x_{\eta }^{\left(\vec{\omega }\right)}\cdot dT \end{gathered}$$

$$-\frac{dS}{k_B}+\frac{d⟨E⟩}{k_{B}T}-\frac{1}{k_{B}T}\sum _{\eta =1}^n{X}_{\eta }d⟨x_{\eta }⟩-\frac{1}{k_{B}T}\sum _{\eta =1}^m{Y}_{\eta }d{y}_{\eta }=0$$

$$\sum _{\vec{\omega }}\frac{\partial }{\partial T}\left(\frac{f_{\vec{\omega }}}{Z}\right)\cdot \left[\log \left(f_{\vec{\omega }}\right)+\frac{E^{\left(\vec{\omega }\right)}}{k_{B}T}-\sum _{\eta =1}^n\frac{X_{\eta }{x}_{\eta }^{\left(\vec{\omega }\right)}}{k_{B}T}\right]dT=0$$

$$\log \left(f_{\vec{\omega }}\right)+\frac{E^{\left(\vec{\omega }\right)}}{k_{B}T}-\sum _{\eta =1}^n\frac{X_{\eta }{x}_{\eta }^{\left(\vec{\omega }\right)}}{k_{B}T}=0$$

$$f_{\vec{\omega }}=\exp \left[\sum _{\eta =1}^n\frac{X_{\eta }{x}_{\eta }^{\left(\vec{\omega }\right)}}{k_{B}T}-\frac{E^{\left(\vec{\omega }\right)}}{k_{B}T}\right]$$

$$\sum _i{E}_{i}d{p}_i=\text{mathchar}22dQ=TdS=-k_{B}T\sum _i\log p_{i}d{p}_i$$

Full text

I Introduction

There are two well known ways to derive the Boltzmann distribution: the micro-canonical derivation, and the maximum entropy principle derivation. Both are found in textbooks, such as Kardar (2007); Callen (1998); chandler1987introduction; Landau and Lifshitz (2013); Balescu (1991) for the micro-canonical derivation and Chandler (1987); Callen (1998); Tolman (1979) for the maximum entropy principle derivation. These two derivations could be naturally extended to derive the generalized Boltzmann distributions for other ensembles such as grand canonical or isothermal - isobaric ensemble. Beyond these two standard textbook derivations, the Boltzmann distribution can also be derived based on quantum dynamicsTasaki (1998).

Although modern statistical thermodynamics dates back to as early as BoltzmannCercignani (1998) and GibbsGibbs (2014), new insights are still being obtained, such as the Jarzynski equalityJarzynski (1997) and its quantum extensionMukamel (2003), as well as fluctuation theoremsCrooks (1999); Evans et al. (1993); Seifert (2005); Esposito and Van den Broeck (2010). There is also interest in entropy and its relationship with information, such as Cerf and Adami (1997); Lieb and Yngvason (2013); Parrondo et al. (2015). In this paper, we study the basic question of the relationship between the generalized Boltzmann distribution, the thermodynamic entropy, and the Gibbs-Shannon entropy.

For context, we start by revisiting the two textbook derivations of the Boltzmann distribution.

I.1 The micro-canonical derivation

The micro-canonical derivation constructs an ensemble with fixed total energy $E$ composed of the system of interest and a reservoir. The fundamental postulate of statistical mechanics states that the probability distribution of allowed micro-states is uniform:

[TABLE]

where $E_{k}$ is the energy of micro-state $k$ and $\Omega_{E}$ is the number of micro-states with energy $E$. The interaction between the system and reservoir is assumed to be weak, in the sense that for a fixed micro-state $i$ of the system with energy $E_{i}$, the reservoir is a micro-canonical ensemble with energy $E_{j}=E-E_{i}$. The assumption of weak interaction also implies that is possible to enumerate the states of the system and reservoir independently. The probability of micro-state $i$ of the system of interest is therefore obtained by marginalizing over the allowed states $j$ of the reservoir

[TABLE]

where $\Omega_{\text{r}(E-E_{i})}$ is the number of states of the reservoir that satisfy $E_{j}=E-E_{i}$, which can be written as

[TABLE]

using the definition of entropy for the micro-canonical ensemble. Since the energy of the system $E_{s}$ is a small fraction of the total energy, we can expand $S_{r}$ around $E$ as a power series of the system energy $E_{s}$ as

[TABLE]

where $E_{r}=E-E_{i}$ is the energy of the reservoir and $T$ is the temperature of the reservoir. This equation becomes exact in the limit of an infinite reservoir at constant temperature T Callen (1998). Combining (LABEL:micrpi), (LABEL:Omegar-Sr) and (LABEL:Sr-expand), we get $\Omega_{\text{r}(E-E_{i})}\propto\exp\left(-\frac{E_{i}}{k_{B}T}\right)$, or, equivalently, $p_{i}\propto\exp\left(-\frac{E_{i}}{k_{B}T}\right)$, which is the Boltzmann distribution.

It is worth mentioning that the assumption of weak interaction between the bath and the system can be relaxed by canonical typicalityGoldstein et al. (2006); Reimann (2007).

I.2 The maximum entropy principle derivation

The maximum entropy principleJaynes (1957a, b) derives the Boltzmann distribution by maximizing the Gibbs-Shannon entropyCover and Thomas (2012) $H=-\sum_{i}p_{i}\log p_{i}$ under the constraints of $\left\langle E\right\rangle=\sum_{i}p_{i}E_{i}$ being a constant $E_{0}$ and of $\sum_{i}p_{i}=1$. The derivation is done using the Lagrangian multiplier method which maximizes the target function

[TABLE]

where $\alpha$ and $\beta$ are both Lagrangian multipliers. Zeroing the derivative of $\mathcal{L}$ with respect to $p_{i}$ gives

[TABLE]

This optimization process has been shown to be equivalent to the micro-canonical derivation as shown in subsection I.1 by applying the maximum entropy principle to the whole isolated system containing the system of interest and reservoir, in a two step fashionLee and Pressé (2012).

I.3 Connecting the canonical ensemble with thermodynamics

After obtaining the probability distribution of the ensemble, we need to establish the connection between thermodynamic variables and ensemble quantities. For the canonical ensemble, the thermodynamic temperature $T$, volume $V$, and particle number $N$ are naturally mapped to the $T,V,N$ parameter of the ensemble. Since the energy of the ensemble is not a parameter but instead a random variable, the mapping of the thermodynamic internal energy $U$ is not as obvious as it is for $T,V,N$.

One then introduce a new postulate that equates the thermodynamic energy $U$ with the ensemble average of the random variable $E$ of system energy:

[TABLE]

where $Z=\sum_{i}e^{-\beta E_{i}}$ is the partition function and $\beta=\frac{1}{k_{B}T}$. Since $d\left(\beta F\right)=Ud\beta-\beta pdV$, i.e. $\beta F$ has natural variables $\beta$ and $V$, and $U=\frac{\partial\left(\beta F\right)}{\partial\beta}$. Comparing this relation with (LABEL:u-is-avg-e) immediately gives the equation for Helmholtz free energy $F=-k_{B}T\log Z$. From the definition of Helmholtz free energy $F=U-TS$, we immediately obtain that the thermodynamic entropy can then be computed as

[TABLE]

where $H=-\sum_{i}p_{i}\log p_{i}$ is the Gibbs-Shannon entropy.

I.4 Our contribution

The above logic shows that

[TABLE]

while we will now prove the opposite direction:

[TABLE]

The contribution of this paper is therefore two fold: (1) shows that the generalized Boltzmann distribution is the only distribution in which the Gibbs-Shannon entropy equals the thermodynamic entropy, and (2) presents a new way of deriving the generalized Boltzmann distribution.

II Main result

We consider a thermodynamic system with $m+n$ generalized forces and coordinatesKubo (1968). The thermodynamic first law can be written as

[TABLE]

We use $U,\chi_{1},\ldots\chi_{n}$ to denote thermodynamic quantities and $E,x_{1},\ldots,x_{n}$ to denote random variables. We also intentionally choose different letters for generalized forces ($X_{i}$ and $Y_{i}$) and displacements ($\chi_{i}$ and $y_{i}$) to emphasize that the ensemble we are going to study is parameterized by $T$, generalized forces $X_{1},\ldots,X_{n}$, and generalized displacements $y_{1}\ldots,y_{m}$; that is, $E,x_{1}\ldots x_{n}$ are random variables, while $T,X_{1},\ldots,X_{n},y_{1},\ldots,y_{n}$ are parameters. In other word, we are studying a $(T,X_{1},\ldots,X_{n},y_{1},\ldots,y_{n})-$ ensemble.

The above notation is a general framework for all thermodynamic systems. For example, for an $(\mu,V,T)-$ ensemble (one component grand canonical ensemble), we have $m=n=1$, $y_{1}=V$, $Y_{1}=-p$, $\chi_{1}=N$, $X_{1}=\mu$.

The main result we present in this paper is the following theorem:

Theorem 1**.**

Consider thermodynamic systems whose first law reads like (LABEL:first-law). Any $\left(T,X_{1},\ldots,X_{n},y_{1},\ldots,y_{m}\right)$ ensemble that has:

Probability density proportional to some function of the ensemble parameters $T,X_{1},\ldots,X_{n},y_{1},\ldots,y_{m}$ and random variables $E,x_{1},x_{2},\cdots,x_{n}$. 2. 2.

$\left\langle E\right\rangle=U$, $\left\langle x_{\eta}\right\rangle=\chi_{\eta}$ for all $\eta=1,\ldots,n$ 3. 3.

The thermodynamic entropy $S$ and the Gibbs-Shannon entropy $H$ have a relationship $S=k_{B}H$

the probability density function of micro-state $\vec{\omega}$ must have the form

[TABLE]

where $E^{\left(\vec{\omega}\right)},x_{1}^{\left(\vec{\omega}\right)},\ldots,x_{n}^{\left(\vec{\omega}\right)}$ are the value of random variables for the state $\vec{\omega}$.

Proof.

Let’s begin by assuming the probability density of micro-state $\vec{\omega}$ is expressed as

[TABLE]

the normalization constant is

[TABLE]

where the sum is over all microscopic states and $f_{\vec{\omega}}$ is short for $f\left(E^{\left(\vec{\omega}\right)},x_{1}^{\left(\vec{\omega}\right)},\ldots,x_{n}^{\left(\vec{\omega}\right)};T,X_{1},\ldots,y_{1},\ldots\right)$. In the above formula, the temperature $T$ appears as a parameter, and micro-states $\vec{\omega}$ and its macro-states $E^{\left(\vec{\omega}\right)},x_{1}^{\left(\vec{\omega}\right)},\ldots,x_{n}^{\left(\vec{\omega}\right)}$ do not depend on $T$. Note that if $S=k_{B}H$, then the thermodynamic entropy

[TABLE]

If we perturb the temperature $T$ by $dT$ and keep the other parameters ($X_{1},\ldots,X_{n},y_{1},\ldots,y_{m}$) fixed, then the change of entropy

[TABLE]

where

[TABLE]

Also

[TABLE]

and therefore

[TABLE]

The same argument applies for all $\left\langle x_{\eta}\right\rangle$, and we therefore have

[TABLE]

By substituting equations in condition 2 of the theorem into (LABEL:first-law), the first law of thermodynamics can be rewritten as

[TABLE]

substituting (LABEL:dSdT), (LABEL:pSpT), (LABEL:dEdT), (LABEL:dxdT) and $dy_{\eta}=0$222Because we are perturbing $T$ while keeping $y_{\eta}$ constant, we have

[TABLE]

Physics laws should be universal, i.e. the above equation must hold for arbitrary system, the only way for this to happen is

[TABLE]

that is

[TABLE]

We therefore obtained the generalized Boltzmann distribution. ∎

III Examples

The formulation of our main result is quite general. In this section, we will show how commonly used ensembles fit into our general framework.

The single component canonical ensemble, i.e. the $(N,V,T)$ ensemble, has $y_{1}=N$, $y_{2}=V$, $Y_{1}=\mu$, $Y_{2}=-p$. There are no $\chi$, $X$ or $x$. The first law reads $dU=TdS-pdV+\mu dN$ and the distribution is $\Pr\left(\vec{\omega}\right)\propto\exp{\left(-\frac{E^{\left(\vec{\omega}\right)}}{k_{B}T}\right)}$.

The two component grand canonical ensemble, i.e. the $(\mu_{1},\mu_{2},V,T)$ ensemble, has $y_{1}=V$, $Y_{1}=-p$, $\chi_{1}=N_{1}$, $\chi_{2}=N_{2}$, $X_{1}=\mu_{1}$, $X_{2}=\mu_{2}$, $x_{1}^{\left(\vec{\omega}\right)}=N_{1}^{\left(\vec{\omega}\right)}$, and $x_{2}^{\left(\vec{\omega}\right)}=N_{2}^{\left(\vec{\omega}\right)}$. The first law reads $dU=TdS-pdV+\mu_{1}dN_{1}+\mu_{2}dN_{2}$ and the distribution is $\Pr\left(\vec{\omega}\right)\propto\exp{\left(\frac{\mu_{1}N_{1}^{\left(\vec{\omega}\right)}+\mu_{2}N_{2}^{\left(\vec{\omega}\right)}-E^{\left(\vec{\omega}\right)}}{k_{B}T}\right)}$.

The single component isothermal–isobaric ensemble, i.e. the $(N,p,T)$ ensemble, has $y_{1}=N$, $Y_{1}=\mu$, $\chi_{1}=V$, $X_{1}=-p$, and $x_{1}^{\left(\vec{\omega}\right)}=V^{\left(\vec{\omega}\right)}$. The first law reads $dU=TdS-pdV+\mu dN$ and the distribution is $\Pr\left(\vec{\omega}\right)\propto\exp{\left(-\frac{pV^{\left(\vec{\omega}\right)}+E^{\left(\vec{\omega}\right)}}{k_{B}T}\right)}$.

IV Discussion

We showed that the ensemble for a non-isolated thermodynamic system at equilibrium described by temperature, parameters $y_{i}$ and expectation values $\left\langle x_{i}\right\rangle$, and whose thermodynamic entropy is given by the Gibbs-Shannon formula, is necessarily described by the generalized Boltzmann’s distribution. It follows that the generalized Boltzmann distribution is the only distribution for which the Gibbs-Shannon entropy equals the thermodynamic entropy.

Unlike familiar thermodynamic quantities such as energy, volume, and pressure, the concept of thermodynamic entropy is harder to grasp. The entropy is nevertheless a well-defined state function of systems at equilibrium, stemming from the fact that $\oint{\mathchar 22\relax\mkern-12.0mud}Q/T=0$ for any closed thermodynamic path along equilibrium states. Conversely, the thermodynamic entropy is defined only for states at equilibriumLieb and Yngvason (2013). So, while the question of what is the form of the entropy for states out of equilibrium is fundamentally ill-posed, it is tempting to try to extend the concept of thermodynamic entropy to stable non-equilibrium systemsParrondo et al. (2015), such as systems at steady state. Our result implies that the entropy of such systems cannot be described by the Gibbs-Shannon formula unless their distribution of states follows the generalized Boltzmann’s distribution.

Boltzmann famously defined the entropy of an isolated system at energy $E$ as $S(E)=-k_{B}\log{p(E)}$ Cercignani (1998), $S(E)$ is the micro-canonical entropy and where $p(E)=1/\Omega$ is the probability of occupancy of any of the degenerate micro-states of the system. The identification of the thermodynamic entropy of a system in equilibrium with a reservoir with the ensemble average of the micro-canonical entropy ($S=-k_{B}\left\langle\log p\right\rangle=-k_{B}\sum_{i}p_{i}\log{p_{i}}$), as in the Gibbs-Shannon entropy formulaGibbs (2014), while known to be consistent with the Boltzmann’s distribution, appears arbitrary. For example, it is worth considering whether there could be another distribution whose Gibbs-Shannon entropy can be identified with the thermodynamic entropy. Our result shows that within the very general thermodynamic assumptions ((LABEL:first-law,_propto)) this is not possible.

It is also worth mention that, for the special case of $(N,V,T)$ ensemble, instead of starting from the thermodynamic first law in (LABEL:first-law), similar results could also be obtainedMa starting from the expression of heat of quantum thermodynamicsMa et al. (2017); Quan et al. (2005); Su et al. (2018); Quan et al. (2007), i.e. ${\mathchar 22\relax\mkern-12.0mud}Q=\sum_{i}E_{i}dp_{i}$: Under the condition $S=k_{B}H$:

[TABLE]

which immediately gives $p_{i}\propto\exp{\left(-\frac{E_{i}}{k_{B}T}\right)}$.

V Acknowledgement

The authors would like to thank Vinícius Wilian D. Cruzeiro for proofreading of this paper, suggestions on the organization of this paper, and technical discussion on the condition of our main results. Discussions with Daniel Zuckerman, John Chodera, and Mike Gilson, have enriched this manuscript.

The authors would also like to thank Yuhan Ma for the discussion on the relationship of the main result of this paper with thermal equilibrium, as well as the connection of the proof of the main result with respect to quantum thermodynamics.