The Poincar\'e-Boltzmann Machine: from Statistical Physics to Machine Learning and back

TL;DR

This paper introduces a topological framework using information cohomology to analyze statistical dependencies in complex systems, bridging statistical physics and machine learning, and providing new insights into multivariate information structures.

Contribution

It develops a cohomological approach to quantify statistical dependencies and independence without relying on traditional assumptions, linking information theory with topological and physical concepts.

Findings

Cohomology quantifies statistical dependences and independence.

Topological approach avoids assumptions of i.i.d. variables.

Introduces free energy landscape for multivariate information.

Abstract

This paper presents the computational methods of information cohomology applied to genetic expression in and in the companion paper and proposes its interpretations in terms of statistical physics and machine learning. In order to further underline the Hochschild cohomological nature af information functions and chain rules, following, the computation of the cohomology in low degrees is detailed to show more directly that the $k$ multivariate mutual-informations (I_k) are k-coboundaries. The k-cocycles condition corresponds to I_k=0, generalizing statistical independence. Hence the cohomology quantifies the statistical dependences and the obstruction to factorization. The topological approach allows to investigate information in the multivariate case without the assumptions of independent identically distributed variables and without mean field approximations. We develop the computationally tractable subcase of simplicial information cohomology represented by entropy H_k and information I_k landscapes and their respective paths. The I_1 component defines a self-internal energy U_k, and I_k,k>1 components define the contribution to a free energy G_k (the total correlation) of the k-body interactions. The set of information paths in simplicial structures is in bijection with the symmetric group and random processes, provides a trivial topological expression of the 2nd law of thermodynamic. The local minima of free-energy, related to conditional information negativity, and conditional independence, characterize a minimum free energy complex. This complex formalizes the minimum free-energy principle in topology, provides a definition of a complex system, and characterizes a multiplicity of local minima that quantifies the diversity observed in biology. I give an interpretation of this complex in terms of frustration in glass and of Van Der Walls k-body interactions for data points.

Equations

$$H_1=H(X_j;P_{X_j})=k\sum _{x\in [N_j]}p(x)\ln p(x)=k\sum _{i=1}^{N_j}{p}_i\ln p_i$$

$$\begin{array}{rl}D(p(x)||q(x))& =D(X;p(x)||q(x))=k\sum _{x\in \mathcal{X}}p(x)\ln \frac{q(x)}{p(x)}\\ & =H(X;p(x),q(x))-H(X;p(x))\end{array}$$

$$\begin{array}{rl}{H}_k& =H(X_1,...,X_k;P_{X_1,...,X_k})\\ & =k\sum _{x_1,...,x_k\in [N_1\times ...\times N_k]}^{N_1\times ...\times N_k}p(x_1.....x_k)\ln p(x_1.....x_k)\\ & =k\sum _{i,j,...,k}^{N_1,...,N_k}{p}_{\underset{\text{k indices}}{\underbrace{ij...k}}}\ln p_{ij...k}\end{array}$$

$$I(X_1;X_2;P_{X_1,X_2})=k\sum _{x_1,x_2\in [N_1\times N_2]}^{N_1\times N_2}p(x_1.x_2)\ln \frac{p(x_1)p(x_2)}{p(x_1.x_2)}$$

$$I_k=I(X_1;...;X_k;P)=k\sum _{x_1,...,x_k\in [N_1\times ...\times N_k]}^{N_1\times ...\times N_k}p(x_1.....x_k)\ln \frac{{\prod }_{I\subset [k];card(I)=i;i\text{odd}}{p}_I}{{\prod }_{I\subset [k];card(I)=i;i\text{even}}{p}_I}$$

$$I_3=k\sum _{x_1,x_2,x_3\in [N_1\times N_2\times N_3]}^{N_1\times N_2\times N_3}p(x_1.x_2.x_3)\ln \frac{p(x_1)p(x_2)p(x_3)p(x_1.x_2.x_3)}{p(x_1.x_2)p(x_1.x_3)p(x_2.x_3)}$$

$$\begin{array}{rl}{C}_k& =C_k(X_1;...X_k;P)=\sum _{i=1}^{k}H(X_i)-H(X_1;...X_k)=\sum _{i=2}^k(-1{)}^i\sum _{I\subset [n];card(I)=i}{I}_i(X_I;P)\\ & =k\sum _{x_1,...,x_k\in [N_1\times ...\times N_k]}^{N_1\times ...\times N_k}p(x_1....x_k)\ln \frac{p(x_1...x_k)}{p(x_1)...p(x_k)}\end{array}$$

$$\begin{gathered} X_2.H_1=H(X_1|X_2;P)=k\sum _{x_1,x_2\in [N_1\times N_2]}^{N_1\ast N_2}p(x_1.x_2)\ln p_{x_2}(x_1) \\ =k\sum _{x_2\in {\mathcal{X}}_2}^{N_2}p(x_2).\left(\sum _{x_1\in {\mathcal{X}}_1}^{N_1}{p}_{x_2}{x}_1\ln p_{x_2}{x}_1\right) \end{gathered}$$

$$X_3.I_2=I(X_1;X_2|X_3;P)=k\sum _{x_1,x_2,x_3\in [N_1\times N_2\times N_3]}^{N_1\times N_2\times N_3}p(x_1.x_2.x_3)\ln \frac{p_{x_3}(x_1)p_{x_3}(x_2)}{p_{x_3}(x_1,x_2)}$$

$$H(X_1;X_2;P)=H(X_1;P)+X_1.H(X_2;P)=H(X_2;P)+X_2.H(X_1;P)$$

$$I(X_1;X_2;P)=H(X_1;P)-X_2.H(X_1;P)=H(X_2;P)-X_1.H(X_2;P)$$

$$H(X_1;...;\widehat{X_i};...;X_{k+1};P)=H(X_1;...;X_{k+1};P)-(X_1;...;\widehat{X_i};...;X_{k+1}).H(X_i;P)$$

$$I(X_1;...;\widehat{X_i};...;X_{k+1};P)=I(X_1;...;X_{k+1};P)+X_i.I(X_1;...;\widehat{X_i};...;X_{k+1};P)$$

$$H_k=H(X_1,...,X_k;P)=\sum _{i=1}^k(X_1,...,X_{i-1}).H(X_i;P)$$

$$I_k=I(X_1;...;X_k;P)=I(X_1)-\sum _{i=2}^k{X}_i.I(X_1;...;X_{i-1})$$

$$H_n(X_1,...,X_n;P)=\sum _{i=1}^n(-1{)}^{i-1}\sum _{I\subset [n];card(I)=i}{I}_i(X_I;P)$$

$$I_n(X_1;...;X_n;P)=\sum _{i=1}^n(-1{)}^{i-1}\sum _{I\subset [n];card(I)=i}{H}_i(X_I;P)$$

$$0\to X^0\stackrel{{\partial }^0}{\to }{X}^1\stackrel{{\partial }^1}{\to }{X}^2\stackrel{{\partial }^2}{\to }...X^{k-1}\stackrel{{\partial }^{k-1}}{\to }{X}^k$$

$$\begin{array}{rl}({\partial }^k)F(X_1;X_2;...;X_{k+1};P)=& X_1.F(X_2;...;X_{k+1};P)\\ & +\sum _{i=1}^k(-1{)}^{i}F(X_1;X_2;...;(X_i,X_{i+1});...;X_{k+1};P)\\ & +(-1{)}^{k+1}F(X_1;...;X_k;P)\end{array}$$

$$\begin{array}{rl}({\partial }_t^k)F(X_1;X_2;...;X_{k+1};P)=& F(X_2;...;X_{k+1};P)\\ & +\sum _{i=1}^k(-1{)}^{i}F(X_1;X_2;...;(X_i,X_{i+1});...;X_{k+1};P)\\ & +(-1{)}^{k+1}F(X_1;...;X_k;P)\end{array}$$

$$\begin{array}{rl}({\partial }_{\ast }^k)F(X_1;X_2;...;X_{k+1};P)=& X_1.F(X_2;...;X_{k+1};P)\\ & +\sum _{i=1}^k(-1{)}^{i}F(X_1;X_2;...;(X_i,X_{i+1});...;X_{k+1};P)\\ & +(-1{)}^{k+1}{X}_{k+1}.F(X_1;...;X_k;P)\end{array}$$

$${\partial }^{2k}F=-I_{2k+1}\text{ and }{\partial }_{\ast }^{2k}F=-{\partial }_t^{2k}F=0$$

$${\partial }^{2k-1}F=0\text{ and }{\partial }_{\ast }^{2k-1}F=-{\partial }_t^{2k}F=-I_{2k}$$

$$U(X_1,...,X_n;P_N)=\sum _{i=1}^n{I}_1(X_i;P_N)$$

$$G(X_1,...,X_n;P_N)=\sum _{i=2}^n(-1{)}^{i-1}\sum _{I\subset [n];card(I)=i}{I}_i(X_I;P_N)=C_n(X_1;...X_n;P_N)$$

$$H_n(X_1,...,X_n;P_N)=U(X_1,...,X_n;P_N)-G(X_1,...,X_n;P_N)$$

$$p(X_1=x_1,...,X_n=x_n)=p_{\underset{\text{n indices}}{\underbrace{ij...n}}}=\frac{1}{Z}{e}^{-\beta E_{ij...n}/k_{B}T}$$

$$H(X_1,...,X_n)/k=\sum _{i,j,..,n}^{N_1.N_2...N_n}{p}_{ij...n}{E}_{ij...n}/k_{B}T+\sum _{i,j,..,n}^{N_1.N_2...N_n}{p}_{ij...n}\ln Z=(⟨E⟩-G)/k_{B}T$$

$$k_{B}T\ln 2.H(X_1,...,X_n)=⟨E⟩-G=U-G$$

$$\sum _{k=1}^{n}H(X_k)=\sum _{i,j,..,n}^{N_1.N_2...N_n}{p}_{ij...n}{E}_{ij...n}$$

Full text

”Now what is science? …it is before all a classification, a manner of bringing together facts which appearances separate, though they are bound together by some natural and hidden kinship. Science, in other words, is a system of relations. …it is in relations alone that objectivity must be sought. …it is relations alone which can be regarded as objective. External objects… are really objects and not fleeting and fugitive appearances, because they are not only groups of sensations, but groups cemented by a constant bond. It is this bond, and this bond alone, which is the object in itself, and this bond is a relation.”

H. Poincaré

1. 1 Introduction

2. 2 Information cohomology

3. 2.1 A long march through information topology

4. 2.2 Information functions (definitions)

5. 2.3 Information structures and coboundaries

6. 2.3.1 first degree (k=1)

7. 2.3.2 Second degree (k=2)

8. 2.3.3 Third degree (k=3)

9. 2.3.4 Higher degrees

10. 3 Simplicial information cohomology

11. 3.1 Simplicial substructures of information

12. 3.2 Topological Self and Free Energy of k-body interacting system - Poincaré-Shannon Machine

13. 3.2.1 Topological Self and Free Energy of k-body interacting system

14. 3.3 k-Entropy and k-Information landscapes

15. 3.4 Information Paths and Minimum Free Energy Complex

16. 3.4.1 Information Paths (definition)

17. 3.4.2 Derivatives, inequalities and conditional mutual information negativity

18. 3.4.3 Information paths are random processes: topological 2nd law of thermodynamic and entropy rate

19. 3.4.4 Local minima and critical dimension

20. 3.4.5 Sum over paths and mean information path

21. 3.4.6 Minimum free energy complex

22. 4 Discussion

23. 4.1 Complexity through finite dimension non-extensivity and non iid

24. 4.2 Epigenetic topological learning - biological diversity

1 Introduction

The present paper has no deep mathematical or novelty pretension and just aims to present some computational aspects and applications to data of the simplicial simplest special case of the cohomology theory developed in [13, 16] and extended by Vigneaux [133, 135] to Tsallis entropies and differential entropy (etc.). The direct application to genetic expression measures, performance in cell type or gene module detection and its relevance in epigenetic co-regulation and differentiation terms can be find in [125]. In its application to empirically measured data, information cohomology is at the cross-road of both data analysis and statistical physics, and I aim to give some keys of its interpretation within those two fields, what could be quoted as ”passing the information between disciplines” in reference to Mezard’s review [95].

Unsupervised neural network learning: the original work based on spin networks by Hopfield [59] formalized fully recurrent networks as $n$ binary random variables ($N=2$). Ackley, Hinton and Sejnowski [1] followed up by imposing the Markov Field condition, allowing the introduction of conditional independence to handle network structures with hidden layers and hidden nodes. The result, the Boltzmann or Helmholtz machine [36], relies on the maximum entropy or free-energy minimization principle, and originally on minimizing the relative entropy between the network and environmental states [1]. The informational approach of topological data analysis provides a direct probabilistic and statistical analysis of the structure of a dataset which allows to bridge the gap with neural networks analysis, and may be to go a step beyond their formalization in mathematic. Indeed, as proposed in [12] and references therein (notably see the whole opus of Marcolli with application to linguistic), there are some reasons to believe that artificial or natural cognition could be homological by nature. Considering neuron as binary random variables provides in the present context an homologically constrained approach of those neural networks where the first input layer is represented by the marginal (single variable, degree 1 component) while hidden layers are associated to higher degrees. In a very naive sense higher cohomological degrees distinguish higher order patterns (or higher dimensional patterns in the simplicial case), just as receptive fields of neural networks recognize higher order features when going to higher depth-rank of neural layers as described in David Marr’s original sketch [88], and now implemented efficiently in deep network structures. Here, in order to allow a direct interpretation of the information topology in standard terms of machine learning; I provide an energy functional interpretation of Mutual Information $I_{k}$ and of Total Correlation $G_{k}$ functions. Notably, the introduction of this multiplicity of ”energy functions” broaden common unsupervised methods in neural network : instead of a single energy and associated gradient descent, Mutual informations provide a combinatorial family of analytically independent functions with independent gradients (see [16] for associated theorems). The scope is restricted to unsupervised learning here, but the supervised subcase with explicit homological information version of the back propagation chain rule, can be readily inferred from this and will be detailed with data analysis examples in future publication and illustrated here [15]. Notably, the mathematical depth of back-propagation algorithm [68, 38, 114] comes from the fact that it implements the chain rule for derivation with respect to the parameters, allowing to learn in the class of differentiable functions. In the information cohomological reformulation, the chain rule becomes the chain rule of mutual-information that are coboundaries (commonly differential operator), parameters are provided by the label-teaching variables.

Statistical physics: On the

Contents

1. 1 Introduction

2. 2 Information cohomology

3. 2.1 A long march through information topology

4. 2.2 Information functions (definitions)

5. 2.3 Information structures and coboundaries

6. 2.3.1 first degree (k=1)

7. 2.3.2 Second degree (k=2)

8. 2.3.3 Third degree (k=3)

9. 2.3.4 Higher degrees

10. 3 Simplicial information cohomology

11. 3.1 Simplicial substructures of information

12. 3.2 Topological Self and Free Energy of k-body interacting system - Poincaré-Shannon Machine

13. 3.2.1 Topological Self and Free Energy of k-body interacting system

14. 3.3 k-Entropy and k-Information landscapes

15. 3.4 Information Paths and Minimum Free Energy Complex

16. 3.4.1 Information Paths (definition)

17. 3.4.2 Derivatives, inequalities and conditional mutual information negativity

18. 3.4.3 Information paths are random processes: topological 2nd law of thermodynamic and entropy rate

19. 3.4.4 Local minima and critical dimension

20. 3.4.5 Sum over paths and mean information path

21. 3.4.6 Minimum free energy complex

22. 4 Discussion

23. 4.1 Complexity through finite dimension non-extensivity and non iid

24. 4.2 Epigenetic topological learning - biological diversity

side of statistical physics, the generalization of binary neuronal-ising models to arbitrary multivalued variables with $N_{i}$ values corresponds to the generalization of spin models to Potts models [143]. Such generalization is pertinent for biological systems in general, since coding and meaningful variables in biology are usually non-digital and only in quite exceptional cases binary. More interestingly, Mutual-Information negativity, a phenomenon known to provide the signature of frustrated states in glasses since the work of Matsuda [89], is related here, in the context of the more general conditional mutual information, to a kind of first order transition, yet seen topologically as the critical points of a simplicial complex. We propose to analyse the statistical interactions quantified by Mutual-Informations in full analogy with the Van Der Walls interaction model while leaving any homogeneous or mean-field assumptions on the variable. This observation makes coincide two phenomena, condensation in statistical physics and clustering in data science. We hence further suggest that the Mutual-Informations functions and homology should be peculiarly useful in the study of frustrated states and other related problems (see Mezard and Montanari [96]), and for data points clustering.

Topological Data Analysis: On the side of applied algebraic topology, the identification of the topological structures of dataset has motivated important researches following the development of persistent homology [81, 44, 28]. Combining in a single framework statistical and topological structures remains an active challenge of data analysis that already gave some interesting results [102, 27]. Some recent works have proposed information theoretical approaches grounded on homology, defining persistent entropy [32, 93], graph’s topological entropy [122], spectral entropy [83], or multilevel Integration entropies [84]. Although the present work is formally different and arose independently of persistence methods, it is possible to provide an intuitive non-rigorous interpretation in persistence terms. Most of the persistent methods consist in approximating the birth and death of the Betti’s numbers of the C̆ech complex obtained by considering balls around each data point while the radius of the ball grows. The C̆ech complex is given by the intersection of the balls, and for combinatorial computational reasons, most of the algorithm restrict to pairwise intersections giving the Vietoris-Ripps complex as an approximation of the C̆ech complex. Our method focuses on intersection of random variables rather than balls around data points: a theorem of Hu Kuo Ting [60] (recalled in [16]) shows the equivalence of mutual information functions with set theoretic finite measurable functions endowed with the intersection operator, formalizing the naive Venn diagram usual interpretation of mutual informations. Hence, leaving the mathematical rigor to allow an interpretation of the current algorithm in the common language of persistence: I compute here an information theoretic analog of the C̆ech complex, and it is not excluded that such analogy can be made formal, and notably to establish some nerve theorem for information structures (see Oudot for review [104]). It turns out that ”zero information intersections” is exactly equivalent to statistical independence (theorem 1.2 [16]). Hence, in regard to current topological data analysis methods, the methods presented here provides an intrinsically probabilistic cohomological framework: the differential operators are fundamental maps in probability-information theory. As a consequence, no metric assumption is required apriori: in practice it is possible to compute the homology for example on position variables and-or on qualitative variable such as ”nice” and ”not nice”. The present method is topological and avoids the introduction a priori of such a metric: rather, a family of Shannon’s pseudometric emerges from the formalism as a first cohomological class (cf. section • ‣ 2.3.1): considering a symmetric action of conditioning, we obtain Shannon’s metric parametrized by a scalar multiplicative constant. Notably, the notion of geodesic used in machine learning is replaced by the homotopical notion of path. On the data analysis side, it provides new algorithm and tools for Topological Data Analysis allowing to rank, detect clusters, functional modules and to make dimensionality reduction; all these classical tasks in data analysis have indeed a direct homological meaning. I propose to call the data analysis methods presented here, the Poincaré-Shannon machine, since it implements simplicial homology and information theory in a single framework, applied effectively to empirical data.

2 Information cohomology

This section provides a short bibliographical note on the inscription of information and probability theory within homological theories 2.1. We also recall the definition of information functions 2.2 and provide a short description of information cohomology computed in the low degrees 2.3, such that the interpretation of entropy and mutual-informations within Hochschild cohomology appears straightforward and clear. There is no new result in this section, but I hope a more simple and helpful presentation for some researchers outside the field of topology, of what can be find in [13, 133, 135] that should be considered for more precise and detailed exposition.

2.1 A long march through information topology

From the mathematical point of view, a motivation of information topology is to capture the ambiguity theory of Galois, which is the essence of group theory or discrete symmetries (see André’s reviews [4, 5]), and Shannon’s information uncertainty theory in a common framework, a path already paved by some results on information inequalities (see Yeung’s results [146]) and in algebraic geometry. In the work of Cathelineau, [30], entropy first appeared in the computation of the degree one homology of the group $SL(2,\mathbb{C})$ with coefficients in the adjoint action by choosing a pertinent definition of the derivative of the Bloch-Wigner dilogarithm. It could be shown that the functional equation with 5-terms of the dilogarithm implies the functional equation of entropy with 4-terms. Kontsevitch [72] discovered that a finite truncated version of the logarithm appearing in cyclotomic studies also satisfied the functional equation of entropy, suggesting a higher degree generalization of information, analog to polylogarithm, and hence showing that the functional equation of entropy holds in p and 0 field characteristics. Elbaz-Vincent and Gangl used algebraic means to construct this information generalization which holds over finite fields [42], and where information functions appear as derivations [43]. After entropy appeared in tropical and idempotent semi-ring analysis in the study of the extension of Witt semiring to the characteristic 1 limit [33], Marcolli and Thorngren developed thermodynamic semiring, and entropy operad that could be constructed as deformation of the tropical semiring [86]. Introducing Rota-Baxter algebras, it allowed to derive a renormalization procedure [85]. Baez, Fritz and Leinster defining the category of finite probability and using Fadeev axiomatization, could show that the only family of functions that has the functorial property is Shannon information loss [9, 11]. Boyom, basing his approach on information and Koszul geometry, developed a more geometrical view of statistical models that notably considers foliations in place of the random variables [25]. Introducing a deformation theoretic framework, and chain complex of random variables, Drumond-Cole, Park and Terilla [39, 40, 105] could construct a homotopy probability theory for which the cumulants coincide with the morphisms of the homotopy algebras. A probabilistic framework, used here, was introduced in [13], and generalized to Tsallis entropies by Vigneaux [133, 135]. The diversity of the formalisms employed in these independent but convergent approaches is astonishing. So, to the question what is information topology, it is only possible to answer that it is under development at the moment. The results of Catelineau, Elbaz-Vincent and Gangl inscribed information into the theory of motives, which according to Beilison’s program is a mixed Hodge-Tate cohomology [18]. All along the development of the application to data, following the cohomology developed by [13, 133] on an explicit probabilistic basis, we aimed to preserve such a structure and unravel its expression in information theoretic terms. Moreover, following Aomoto’s results [7, 53], the actual conjecture [13] is that the higher classes of information cohomology should be some kind of polylogarithmic k-form (k-differential volume that are symmetric and additive, and that correspond to the cocycle conditions for the cohomology of Lie groups [7]). The following developments suggest that these higher information groups should be the families of functions satisfying the functional equations of k-independence $I_{k}=0$, a rather vague but intuitive view that can be tested in special cases.

2.2 Information functions (definitions)

The information functions used in [13] and the present study were originally defined by Shannon [117] and Kullback [74] and further generalized and developed by Hu Kuo Ting [60] and Yeung [147] (see also McGill [92]). These functions include entropy, noted $H_{1}=H(X;P)$, joint entropy, noted $H_{k}=H(X_{1},...,X_{k};P)$, mutual-information noted $I_{2}=I(X_{1};X_{2};P)$, multivariate k-mutual-information, noted $I_{k}=I(X_{1};...;X_{k};P)$ and the conditional entropy and mutual information, noted $Y.H_{k}=H(X_{1},...,X_{k}|Y;P)$ and $Y.I_{k}=I(X_{1};...;X_{k}|Y;P)$. The classical expression of these functions is the following (using $k=-1/\ln 2$, the usual bit unit):

• •

The Shannon-Gibbs entropy of a single variable $X_{j}$ is defined by [117]:

[TABLE]

where $[N_{j}]=\{1,...,N_{j}\}$ denotes the alphabet of $X_{j}$.

• •

The relative entropy or Kullback-Liebler divergence, which was also called ”discrimination information” by Kullback [74], is defined for two probability mass function $p(x)$ and $q(x)$ by:

[TABLE]

where $H(X;p(x),q(x))$ is the cross-entropy and $H(X;p(x))$ the Shannon entropy. It hence generates as a special case minus entropy, taking the deterministic constant probability $q(x)=1$. With the convention $k=-1/\ln 2$, $D(p(x)||q(x))$ is always positive or null.

• •

The joint entropy is defined for any joint-product of $k$ random variables $(X_{1},...,X_{k})$ and for a probability joint-distribution $\mathbb{P}_{(X_{1},...,X_{k})}$ by [117]:

[TABLE]

where $[N_{1}\times...\times N_{k}]=\{1,...,N_{j}\times...\times N_{k}\}$ denotes the alphabet of $(X_{1},...,X_{k})$.

• •

The mutual information of two variables $X_{1},X_{2}$ is defined as [117]:

[TABLE]

And it can be generalized to k-mutual-information (also called co-information) using the alternated sums given by equation 17, as originally defined by McGill [92] and Hu Kuo Ting [60], giving:

[TABLE]

For example, the 3-mutual information is the function:

[TABLE]

For $k\geq 3$, $I_{k}$ can be negative [60].

• •

The total correlation introduced by Watanabe [138], called integration by Tononi and Edelman [130] or multi-information by Studený and Vejnarova [121] and Margolin and colleagues [87], which we note $C_{k}(X_{1};...X_{k};P)$, is defined by:

[TABLE]

For two variables the total correlation is equal to the mutual-information ($C_{2}=I_{2}$). The total correlation has the nice property of being a relative entropy 2 between marginal and joint-variable and hence to be always non-negative.

• •

The conditional entropy of $X_{1}$ knowing (or given) $X_{2}$ is defined as [117]:

[TABLE]

Conditional joint-entropy, $X_{3}.H(X_{1},X_{2})$ or $(X_{1},X_{2}).H(X_{3})$, is defined analogously by replacing the marginal probabilities by the joint probabilities.

• •

The conditional mutual information of two variables $X_{1},X_{2}$ knowing a third $X_{3}$ is defined as [117]:

[TABLE]

Conditional mutual information generates all the preceding information functions as subcases, as shown by Yeung [147]. We have the theorem : if $X_{3}=\Omega$ then it gives the mutual information, if $X_{2}=X_{1}$ it gives conditional entropy, and if both conditions are satisfied, it gives entropy. Notably, we have $I_{1}=H_{1}$.

We now give the few information equalities and inequalities that are of central use in the homological framework, in the information diagrams and for the estimation of the informations from the data.

We have the chain rules (see [35] for proofs):

[TABLE]

[TABLE]

That we can write more generally (where the hat denotes the omission of the variable):

[TABLE]

That we can write in short $H_{k+1}-H_{k}=(X_{1},...X_{k}).H(X_{k+1})$

[TABLE]

That we can write in short $I_{k-1}-I_{k}=X_{k}.I_{k-1}$, generating the chain rule 10 as special case.

These two equations provide recurrence relationships that give an alternative formulation of the chain rules in terms of a chosen path on the lattice of information structures:

[TABLE]

where we assume $H(X_{1};P)=X_{0}.H(X_{1};P)$ and hence that $X_{0}$ is the greatest element $X_{0}=\Omega$.

[TABLE]

We have the alternated sums or inclusion-exclusion rules [60, 89, 13]:

[TABLE]

[TABLE]

For example: $H_{3}(X_{1},X_{2},X_{3})=I_{1}(X_{1})+I_{1}(X_{2})+I_{1}(X_{3})-I_{2}(X_{1};X_{2})-I_{2}(X_{1};X_{3})-I_{2}(X_{2};X_{3})+I_{3}(X_{1};X_{2};X_{3})$

The chain rule of mutual-information goes together with the following inequalities discovered by Matsuda [89]. For all random variables $X_{1};..;X_{k}$ with associated joint probability distribution $P$ we have:

• •

$X_{k}.I(X_{1};..;X_{k-1};P)\geq 0$ if and only if $I(X_{1};..;X_{k-1};P)\geq I(X_{1};..;X_{k};P)$ (in short: $I_{k-1}\geq I_{k}$ )

• •

$X_{k}.I(X_{1};..;X_{k-1};P)<0$ if and only if $I(X_{1};..;X_{k-1};P)<I(X_{1};..;X_{k};P)$ (in short: $I_{k-1}<I_{k}$ )

that fully characterize the phenomenon of information negativity as an increasing or diverging sequence of mutual information.

2.3 Information structures and coboundaries

This section justifies the choice of functions and algorithm, the topological nature of the data analysis and the approximations we had to concede for the computation. In the general formulation of information cohomology, the random variables are partitions of the atomic probabilities of a finite probability space $(\Omega,\mathcal{B},P)$ (e.g. all their equivalence classes). The Joint-Variable $(X_{1},X_{2})$ is the less fine partition that is finer than $X_{1}$ and $X_{2}$; the whole lattice of partitions $\Pi$ [6] corresponds to the lattice of joint random variables [48, 13]. Then, a general information structure is defined to be the triple $(\Omega,\Pi,P)$. A more modern and general expression in category theory and topos is given in [13, 133]. $(X_{1},...,X_{k};P)$ designates the image law of the probability $P$ by the measurable function of joint variables $(X_{1},...,X_{k})$. Figure 1 gives a simple example of the lattice of partitions for 4 atomic probabilities, with the simplicial sublattice used for data analysis. Atomic probabilities are also illustrated in a Figure the associated paper [16].

On this general information structure, we consider the real module of all measurable functions $F(X_{1},...,X_{k};P)$, and the conditioning-expectation by $Y$ of measurable functions as the action of $Y$ on the functional module, noted $Y.F(X_{1},...,X_{k};P)$, such that it corresponds to the usual definition of conditional entropy (equ. 8). We define our complexes of measurable functions of random variables $X^{k}=F(X_{1},...,X_{k};P)$, and the cochain complexes $(X^{k},\partial^{k})$ as :

[TABLE]

where $\partial^{k}$ is the left action co-boundary that Hochschild proposed for associative and ring structures [57]. A similar construction of random variable complex was given by Drumond-Cole, Park and Terilla [39, 40]. We consider also the two other directly related cohomologies that are defined by considering a trivial left action [13] and a symmetric (left and right) action [52, 139, 67] of conditioning:

• •

The left action Hochschild-information coboundary and cohomology (with trivial right action):

[TABLE]

This coboundary, with a trivial right action, is the usual coboundary of Galois cohomology ([126], p.2), and in general it is the coboundary of homological algebra obtained by Cartan and Eilenberg [29] and MacLane [82] (non homogenous bar complex).

• •

The ”topological-trivial” Hochschild-information coboundary and cohomology: considering a trivial left action in the preceding setting , e.g. $X_{1}.F(X_{2};...;X_{k+1})=F(X_{2};...;X_{k+1})$. It is the subset of the preceding case, which is invariant under the action of conditioning. We obtain the topological coboundary $(\partial_{t}^{k})$ [13]:

[TABLE]

• •

The symmetric Hochschild-information coboundary and cohomology: as introduced by Gerstenhaber and Shack [52], Kassel [67] (p.13) and Weibel [139] (chap.9), we consider a symmetric (left and right) action of conditioning, that is $X_{1}.F(X_{2};...;X_{k+1})=F(X_{2};...;X_{k+1}).X_{1}$. The left action module is essentially the same as considering a symmetric action bimodule [52, 67, 139]. We hence obtain the following symmetric coboundary $(\partial_{\ast}^{k})$:

[TABLE]

Based on these definitions, Baudot and Bennequin [13] computed the first homology class in the left action Hochschild-information cohomology case and the coboundaries in higher degrees. We introduce here the symmetric case, and detail the higher degree cases by direct specialization of the co-boundaries formulas, such that it appears that information functions and chain rules are homological by nature. For notation clarity, we omit the probability in the writing of the functions, and when specifically stated replace their notation $F$ by their usual corresponding informational function notation $H,I$.

2.3.1 first degree (k=1)

For the first degree $k=1$, we have the following results:

• •

The left 1-co-boundary is $(\partial^{1})F(X_{1};X_{2})=X_{1}.F(X_{2})-F(X_{1},X_{2})+F(X_{1})$. The 1-cocycle condition $(\partial^{1})F(X_{1};X_{2})=0$ gives $F(X_{1},X_{2})=F(X_{1})+X_{1}.F(X_{2})$, which is the chain rule of information shown in equation 10. Then, following Kendall [69] and Lee [78], it is possible to recover the functional equation of information and to characterize uniquely, up to the arbitrary multiplicative constant $k$, the entropy (equation 1) as the first class of cohomology [13, 133]. This main theorem allows us to obtain the other information functions in what follows. Marcolli and Thorngren [86], Leinster, Fritz and Baez [11, 9] obtained independently an analog result using measure-preserving function and characteristic one Witt construction, respectively. In these various theoretical settings, this result extends to relative entropy [86, 11, 13], and Tsallis entropies [86, 133].

• •

The topological 1-coboundary $(\partial_{t}^{1})$ is $(\partial_{t}^{1})F(X_{1};X_{2})=F(X_{2})-F(X_{1},X_{2})+F(X_{1})$, which corresponds to the definition of mutual information $(\partial_{t}^{1})F(X_{1};X_{2})=I(X_{1};X_{2})=H(X_{1})+H(X_{2})-H(X_{1},X_{2})$, and hence $I_{2}$ is a topological 1-coboundary.

• •

The symmetric 1-coboundary $(\partial_{\ast}^{1})$ is $(\partial_{\ast}^{1})F(X_{1};X_{2})=X_{1}.F(X_{2})-F(X_{1},X_{2})+X_{2}.F(X_{1})$, which corresponds to the negative of the pairwise mutual information $(\partial_{\ast}^{1})F(X_{1};X_{2})=X_{2}.H(X_{1})+X_{1}.H(X_{2})-H(X_{1},X_{2})=-I(X_{1};X_{2})$, and hence $-I_{2}$ is a symmetric 1-coboundary. Moreover, the 1-cocycle condition $(\partial_{\ast}^{1})F(X_{1};X_{2})=0$ characterizes functions satisfying $F(X_{1},X_{2})=X_{2}.F(X_{1})+X_{1}.F(X_{2})$, which corresponds to the information pseudo-metric discovered by Shannon [116], Rajski [110], Zurek [150] and Bennett [21], and has further been applied for hierarchical clustering and finding categories in data by Kraskov and Grassberger [73]: $H(X_{1}\bigtriangleup X_{2})=X_{2}.H(X_{1})+X_{1}.H(X_{2})=H(X_{1},X_{2})-I(X_{1};X_{2})$. Therefore, up to an arbitrary scalar multiplicative constant $k$, the information pseudo-metric $H(X_{1}\bigtriangleup X_{2})$ is the first class of symmetric cohomology. This pseudo metric is represented in the Figure . It generalizes to pseudo k-volumes that we define by $V_{k}=H_{k}-I_{k}$ (particularly interesting symmetric functions computed by the provided software).

2.3.2 Second degree (k=2)

For the second degree $k=2$, we have the following results:

• •

The left 2-co-boundary is $\partial^{2}F(X_{1};X_{2};X_{3})=X_{1}.F(X_{2};X_{3})-F((X_{1},X_{2});X_{3})+F(X_{1};(X_{2},X_{3}))-F(X_{1};X_{2})$, which corresponds to minus the 3-mutual information $\partial^{2}F(X_{1};X_{2};X_{3})=X_{1}.I(X_{2};X_{3})-I((X_{1},X_{2});X_{3})+I(X_{1};(X_{2},X_{3}))-I(X_{1};X_{2})=-I(X_{1};X_{2};X_{3})$, and hence $-I_{3}$ is left 2-coboundary.

• •

The topological 2-coboundary is $(\partial_{t}^{2})F(X_{1};X_{2};X_{3})=F(X_{2};X_{3})-F((X_{1},X_{2});X_{3})+F(X_{1};(X_{2},X_{3}))-F(X_{1};X_{2})$, which corresponds in information to $\partial_{t}^{2}F(X_{1};X_{2};X_{3})=I(X_{2};X_{3})-I((X_{1},X_{2});X_{3})+I(X_{1};(X_{2},X_{3}))-I(X_{1};X_{2})=0$, and hence the topological 2-coboundary is always null-trivial.

• •

The symmetric 2-coboundary is $(\partial_{\ast}^{2})F(X_{1};X_{2};X_{3})=X_{1}.F(X_{2};X_{3})-F((X_{1},X_{2});X_{3})+F(X_{1};(X_{2},X_{3}))-X_{3}.F(X_{1};X_{2})$, which corresponds in information to $\partial_{\ast}^{2}F(X_{1};X_{2};X_{3})=X_{1}.I(X_{2};X_{3})-I((X_{1},X_{2});X_{3})+I(X_{1};(X_{2},X_{3}))-X_{3}.I(X_{1};X_{2})=0$, and hence the symmetric 2-coboundary is always null-trivial.

2.3.3 Third degree (k=3)

For the third degree $k=3$, we have the following results:

• •

The left 3-co-boundary is $\partial^{3}F(X_{1};X_{2};X_{3};X_{4})=X_{1}.F(X_{2};X_{3};X_{4})-F((X_{1},X_{2});X_{3};X_{4})+F(X_{1};(X_{2},X_{3});X_{4})-F(X_{1};X_{2};(X_{3},X_{4}))+F(X_{1};X_{2};X_{3})$, which corresponds in information to $\partial^{3}F(X_{1};X_{2};X_{3};X_{4})=X_{1}.I(X_{2};X_{3};X_{4})-I((X_{1},X_{2});X_{3};X_{4})+I(X_{1};(X_{2},X_{3});X_{4})-I(X_{1};X_{2};(X_{3},X_{4}))+I(X_{1};X_{2};X_{3})=0$, and hence the left 3-coboundary is always null-trivial.

• •

The topological 3-coboundary is $\partial_{t}^{3}F(X_{1};X_{2};X_{3};X_{4})=F(X_{2};X_{3};X_{4})-F((X_{1},X_{2});X_{3};X_{4})+F(X_{1};(X_{2},X_{3});X_{4})-F(X_{1};X_{2};(X_{3},X_{4}))+F(X_{1};X_{2};X_{3})$, which corresponds in information to $\partial_{t}^{3}F(X_{1};X_{2};X_{3};X_{4})=I(X_{2};X_{3};X_{4})-I((X_{1},X_{2});X_{3};X_{4})+I(X_{1};(X_{2},X_{3});X_{4})-I(X_{1};X_{2};(X_{3},X_{4}))+I(X_{1};X_{2};X_{3})=I(X_{1};X_{2};X_{3};X_{4})$, and hence $I_{4}$ is a topological 3-coboundary.

• •

The symmetric 3-coboundary is $(\partial_{\ast}^{3})F(X_{1};X_{2};X_{3};X_{4})=X_{1}.F(X_{2};X_{3};X_{4})-F((X_{1},X_{2});X_{3};X_{4})+F(X_{1};(X_{2},X_{3});X_{4})-F(X_{1};X_{2};(X_{3},X_{4}))+X_{4}.F(X_{1};X_{2};X_{3})$, which corresponds in information to $\partial_{\ast}^{3}F(X_{1};X_{2};X_{3};X_{4})=X_{1}.I(X_{2};X_{3};X_{4})-I((X_{1},X_{2});X_{3};X_{4})+I(X_{1};(X_{2},X_{3});X_{4})-I(X_{1};X_{2};(X_{3},X_{4}))+X_{4}.I(X_{1};X_{2};X_{3})=-I(X_{1};X_{2};X_{3};X_{4})$, and hence $-I_{4}$ is a symmetric 3-coboundary.

2.3.4 Higher degrees

For $k=4$, we obtain $\partial^{4}F(X_{1};X_{2};X_{3};X_{4};X_{5})=-I_{5}$, and $\partial_{t}^{5}F(X_{1};X_{2};X_{3};X_{4};X_{5})=0$ , and $\partial_{\ast}^{5}F(X_{1};X_{2};X_{3};X_{4};X_{5})=0$. For arbitrary $k$, the symmetric coboundaries are just the opposite of the topological coboundaries $\partial_{t}^{k}=-\partial_{\ast}^{k}$. It is possible to generalize to arbitrary degrees [13] by remarking that we have:

• •

For even degrees $2k$: we have $I_{2k}=-\partial_{t}I_{2k-1}$ and then $I_{2k}=\partial_{t}\partial\partial_{t}...\partial\partial_{t}H$ with $2k-1$ boundary terms. In conclusion, we have:

[TABLE]

• •

For odd degrees $2k+1$: $I_{2k+1}=-\partial I_{2k-1}$ and then $I_{2k+1}=-\partial\partial_{t}\partial...\partial\partial_{t}H$ with $2k$ boundary terms. In conclusion, we have:

[TABLE]

In [17, 16](theorem 1.2), we show that the mutual independence of n variables is equivalent to the vanishing of all $I_{k}$ functions for all $2\leq k\leq n$. As a probabilistic interpretation and conclusion, the information cohomology hence quantifies statistical dependences at all degrees and the obstruction to factorization. Moreover, $k$-independence coincides with cocycles. We therefore expect that the higher cocycles of information, conjectured to be polylogarithmic forms [13, 43, 42], are characterized by the functional equations $I_{k}=0$, and quantify statistical k-independence.

3 Simplicial information cohomology

3.1 Simplicial substructures of information

The general information structure, relying on the information functions defined on the whole lattice of partitions, encompasses all possible statistical dependences and relations, since by definition it considers all possible equivalent classes on a probability space. One could hence expect this general structure to provide the promising theoretical framework for classification tasks on data, and this is probably true in theory. However, this general case hardly allows any interesting computational investigation as it implies an exhaustive exploration of computational complexity following Bell’s combinatoric in $\mathcal{O}(exp(exp(N^{n}))$) for $n$ $N$-ary variables. This fact was already remarked in the study of aggregation for Artificial Intelligence by Lamarche-Perrin and colleagues [75]. At each order $k$, the number of $k$-joint-entropy and $k$-mutual-information to evaluate is given by Stirling numbers of the second kind $S(n,k)$ that sum to Bell number $B_{n}$, $B_{n}=\sum_{k=0}^{n}S(n,k)$. For example, considering 16 variables that can take 8 values each, we have $8^{16}=2^{48}\approx 3.10^{14}$ atomic probabilities and the partition lattice of variables exhibits around $e^{e^{2^{48}}-1}\geq 2^{200}$ elements to compute. Such computational reef can be decreased by considering the sample-size $m$, which is the number of trials, repetitions or points that is used to effectively estimate the empirical probability. It restricts the computation to $\mathcal{O}(exp(exp(m))$, which remains insurmountable in practice with our current classical Turing machines. To circumvent this computational barrier, data analysis is developed on the simplest and oldest subcase of Hochschild cohomology: the simplicial cohomology, which we hence call the simplicial information cohomology and structure, and which corresponds to a subcase of cohomology and structure introduced previously (see Figure 1b.). It corresponds to the example 1 and 4 in [13]. For simplicity, we note also the simplicial information structure $(\Omega,\Delta^{n},P)$, $\Delta^{n}=(X_{1},...,X_{n};P)$, as we will not come back to the general setting. Joint $(X_{1},X_{2})$ and meet $(X_{1};X_{2})$ operations on random variables are the usual joint-union and meet-intersection of Boolean algebra and define two opposite-dual monoids, generating freely the lattice of all subsets and its dual. The combinatorics of the simplicial information structure follow binomial coefficients and, for each degree $k$ in an information structure of $n$ variables, we have $\binom{n}{k}=\frac{n!}{k!(n-k!)}$ elements that are in one to one correspondence with the $k$-faces (the $k$-tuples) of the $n$-simplex of random variables (or its barycentric subdivisions). It is a (simplicial) substructure of the general structure since any finite lattice is a sub-lattice of the partition lattice [109]. This lattice embedding and the fact that simplicial cohomology is a special case of Hochschild cohomology can also be inferred directly from their coboundary expression and has been explicitly formalized in homology: notably, Gerstenhaber and Shack showed that a functor, noted $\varSigma\mapsto k_{\varSigma}!$, induces an isomorphism between simplicial and Hochschild cohomology $H^{\bullet}(\varSigma,k)\cong H^{\bullet}(k_{\varSigma}!,k_{\varSigma}!)$ [51]. A simplicial complex $X^{k}=F(X_{1},...,X_{k};P)$ of measurable functions is any subcomplex of this simplex $\Delta^{n}$ with $k\leq n$, and any simplicial complex can be realized as a subcomplex of a simplex (see Steenrod [119] p.296). The information landscapes presented in Figure 3 illustrate an example of such a lattice/information structure. Moreover in this ordinary homological structure, the degree obviously coincides with the dimension of the data space (the data space is in general $\mathbb{R}^{n}$, the space of ”co-ordinate” values of the variables). This homological (algebraic, geometric and combinatorial) restriction to the simplicial subcase can have some important statistical consequences. In practice, whereas the consideration of the partition lattice ensured that no reasonable (up to logical equivalence) statistical dependences could be missed (since all the possible equivalence classes on the atomic probabilities were considered), the monoidal simplicial structure unavoidably misses some possible statistical dependences as shown and exemplified by James and Crutchfield [61].

3.2 Topological Self and Free Energy of k-body interacting system - Poincaré-Shannon Machine

3.2.1 Topological Self and Free Energy of k-body interacting system

The basic idea behind the development of topological quantum field theories [8, 142, 115] was to define the action and energy functionals on a purely topological ground, independently of any metric assumptions, and to derive from this the correlation functions or partition functions. Here, in an elementary model for applied purposes, we define, in the special case of classical and discrete probability, the k-mutual-information $I_{k}$ (that generalize the correlation functions to nonlinear relation [111]), as the contribution of the k-body interactions to the energy functional. Some further observations support such a definition: i) as stated in [13] (Th.D), the signed mutual-informations $(-1)^{k}I_{k}$ defining energy are sub-harmonic, a kind of weak convexity ii) in the next sections, we define the paths of information and show that they are equivalent to the discrete symmetry group iii) from the empirical point of view, Figure 5 shows that these energy functionals estimated on real data behave as expected for usual k-body homogeneous formalism such as Van-Der-Walls model, or more refined Density Functional Theory (DFT) [58, 70]. These definitions, given in the context of simplicial structures, generalize to the case of partitions lattice, and altogether provide the usual thermodynamical and machine-learning expressions and interpretation of mutual-information quantities: some new methods free of metric assumptions. There are two qualitatively and formally different components in the $I_{k}$ :

• •

Self-internal information energy (definition): for $k=1$, $I_{1}$ and their sum in an information structure expressed in equation 16, namely $\sum_{T\subset[n];card(T)=1}I_{1}(X_{T};P)$, are a self-interaction component, since it sums over marginal information-entropy $I_{1}(X_{i})=H_{1}(X_{i})$. We call the first dimension mutual information component $U(X_{1},...,X_{n};P_{N})$ the self information or internal information energy, in analogy to usual statistical physics and notably DFT:

[TABLE]

Note that in the present context, which is discrete and where the interactions do not depend on a metric, the self-interaction does not diverge, which is a usual problem with metric continuous formalism and was the original motivation for regularization and renormalization infinite corrections, considered by Feynman and Dirac as the mathematical default of the formalism [47, 37].

• •

k-free-energy and total-free-energy (definition): for $k\geq 2$, $(-1)^{k}I_{k}$ and their sum in an information structure (equation 16) quantify the contribution of the k-body interactions. We call the k${}^{\text{th}}$ dimension mutual information component $(-1)^{k}I_{k}$, given in equation 5, the k-free-information-energy. We call the (cumulative) sum over dimensions of these k-free-information-energies starting at pairwise interactions (dimension 2), the total n-free-information-energy, and note it $G(X_{1},...,X_{n};P_{N})$:

[TABLE]

The total free-energy is the total correlation (equation 7) introduced by Watanabe in 1960 [138] that quantifies statistical dependence in the work of Studený and Vejnarova [121] and Margolin and colleagues [87], and among other examples consciousness in the work of Tononi and Edelman [130]. In agreement with the results of Baez and Pollard in their study of biological dynamics using out-of-equilibrium formalism [10], and the appendix of the companion paper on Bayes Free Energy [16] the total free-energy is a relative entropy. The consideration that free energy is the peculiar case of total correlation within the set of relative entropies, accounts for the fact that the free energy shall be a symmetric function of the variables associated to the various bodies, e.g $f(X;Y)=f(Y;X)$ in the pairwise interaction case. Moreover, whereas $I_{k}$ energy component can be negative, the $G_{k}$ total energy component is always non-negative. Each $(-1)^{k}I_{k}$ term in the free energy can be understood as a free energy correction accounting for the k-body interactions.

Entropy is given by the alternated sums of information (equation 16), which then read as the usual isotherm thermodynamic relation:

[TABLE]

This information theoretic formulation of thermodynamic relation follows Jaynes [63, 62], Landauer [77], Wheeler [141], and Bennett’s[20] original work, and is general in the sense that it is finite and discrete, and holds independently of the assumption of the system being in equilibrium or not, i.e. for whatever finite probability. In more probabilistic terms, it does not assume that the variables are identically distributed, a condition that is required for the application of classical central limit theorems (CLT) to obtain the normal distributions in the asymptotic limit [136]. In the special case where one postulates that the probability follows the equilibrium Gibbs distribution, which is also the maximum entropy distribution [41, 34], the expression of the joint-entropy ($k=-1/\ln 2$) allows to recover the equilibrium fundamental relation, as usually achieved in statistical physics (see Adami and Cerf [3] and Kapranov [66] for more details). Explicitly, let’s consider the Gibb’s distribution:

[TABLE]

where $E_{ij...n}$ is the energy of the elementary-atomic probability $p_{ij...n}$, $k_{B}$ is Boltzmann constant, $T$ the temperature and $Z=\sum_{i,j,..,n}^{N_{i}.N_{j}...N_{n}}e^{-E_{ij...n}/k_{B}T}$ is the partition function, such that $\sum_{i,j,..,n}^{N_{i}.N_{j}...N_{n}}p_{ij...n}=1$. Since $H(X_{1},...,X_{n})=k\sum_{i,j,..,n}^{N_{i}.N_{j}...N_{n}}p_{ij...n}\ln p_{ij...n}$, equals the thermodynamic entropy function $S$ up to the arbitrary Landauer constant factor $k_{B}\ln 2$, $S=k_{B}\ln 2H(X_{1},...,X_{n})$, the entropy for Gibbs distribution gives:

[TABLE]

, which gives the expected thermodynamical relation:

[TABLE]

, where $G$ is the free-energy $G=-k_{B}T\ln Z$.

In the general case of arbitrary random variables (not necessarily iid) and discrete probability space, the identification of marginal informations with internal energy:

[TABLE]

implies by direct algebraic calculus that:

[TABLE]

, where the marginal probability $p_{\bullet\bullet...k...\bullet}$ is the sum over all probabilities for which $X_{k}=x_{k}$. It is hence tempting to identify the elementary-atomic energies $E_{ij...n}$ with the elementary marginal informations $\ln p_{\bullet\bullet...k...\bullet}$. This is achieved uniquely by considering that such an elementary energy function must satisfy the additivity axiom (extensivity): ($E(X_{i}=x_{i},Xj=x_{j})=E_{i,j}=E_{ij}=E_{i}+E_{j}$), which is the functional equation of the logarithm. The original proof goes back at least to Kepler, an elementary version was given by Erdos [45], and in Information theory terms can be found in the proofs of uniqueness of ”single event information function” by Aczel and Darokzy ([2], p.3). It establishes the following proposition:

Theorem 3.1**.**

Given a simplicial information structure, the elementary energies satisfying the extensivity axiom are the functions:

[TABLE]

, where $k$ is an arbitrary constant settled to $k=-1/\ln 2$ for units in bit.

The geometric meaning of these elementary energies as log of marginal elementary probability volumes (locally Euclidean) is illustrated in Figure 2 and further underlines that $I_{k,k\geq 2}$ are volume corrections accounting for the statistical dependences among marginal variables.

Examples: i) in the example of 3 binary random variables ($n=3,N_{1}=N_{2}=N_{3}=2$, three variables of Bernoulli) illustrated in the Figure of the associated paper [16], we have $E_{000}=-\ln(p_{0\bullet\bullet}p_{\bullet 0\bullet}p_{\bullet\bullet 0})$, $E_{000}=-\ln(p_{000}+p_{010}+p_{001}+p_{011})-\ln(p_{000}+p_{100}+p_{001}+p_{101})-\ln(p_{000}+p_{100}+p_{010}+p_{110})$ and in the configuration of negative-entangled-Borromean information of the Figure of the associated paper [16], we obtain $E_{000}=3$ in bit units, and similarly $E_{001}=E_{010}=E_{011}=E_{101}=E_{110}=E_{111}=3$, and we hence recover $U=\sum_{i,j,k}^{8}p_{ijk}E_{ijk}=\sum_{i=1}^{3}H(X_{i})=3$ bits. Note that the limit $0\ln 0\sim 0$ avoids singularity of elementary energies.

ii) in the special case of identically distributed variables, $p_{\bullet\bullet...k...\bullet}=p_{\bullet...j...\bullet\bullet}$, we have $E_{ij...n}=nk\ln p_{\bullet\bullet...k...\bullet}$ and hence the marginal Gibbs distribution: $p_{\bullet\bullet...k...\bullet}=e^{\frac{E_{ij...n}}{nk}}$ .

iii) for independent identically distributed variable (non-interacting), we have $G_{n}=0$, and hence:

[TABLE]

iv) considering the variables to be the $6n$ variables of the phase space, with one variable of position and one variable of momentum per body (noted $(X^{1}_{k},X^{2}_{k},X^{3}_{k},P^{1}_{k},P^{2}_{k},P^{3}_{k})$ for the kth body), it is possible to re-express the semi-classical formalism according to which the entropy formulation is (Landau and Lifshitz [76],p.22):

[TABLE]

It is achieved by identifying the internal and free energy as following:

[TABLE]

[TABLE]

This identifies the elementary volumes/probabilities with the Planck constant, the quantum of action (the consistency in the units is realized in section 3.4.3 by the introduction of time). The quantum of action can be illustrated by considering in the Figure 2 that it is the surface of the square/rectangle for two conjugate variables (considered as position and momentum). In this setting, $\Delta X\Delta P$ quantifies the non-extensivity of the volume in the phase-space due to interactions, or in other words, the mutual-informations account for the consideration of the dependence of the subsystems considered as opened and exchanging energy. As noted by Baez and Pollard, the relative entropy provides a quantitative measure of how far from equilibrium the whole system is [10]. The basic principle of such expression of information theory in physics is known at least since Jaynes’s work [62, 64].

As a conclusion, information topology applies, without imposing metric or symplectic or contact structures, to physical formalism of n-body interacting systems relying on empirical measures. Considering the $3n$ or $6n$ dimensions (degrees of freedom) of a configuration or a phase space as random variables, it is possible to recover the (semi) classical statistical physics formalism. It is also interesting to discuss the status of the analog of the temperature variable in the present formalism which is played by the graining, which is the size $N_{i}$ of the alphabet of a variable $X_{i}$. In usual thermodynamic we have $H(X^{n};P_{N})=T.S(X^{n})$, and to stay consistent, temperature shall be a functional inverse of the graining $N$, lowest temperature being the finest grain (large $N$), highest temperature being the coarsest graining (small $N$).

3.3 k-Entropy and k-Information landscapes

**Information landscapes (definition):

**The information landscapes are a representation of the (semi)-lattice of information structures where each element is represented as a function of its corresponding value of entropy or mutual information. In abscissa are the dimensions $k$ and in ordinate the values of information functions of a given subset of k variables.

In data science terms, these landscapes provide a visualization of the potentially high-dimensional structure of the data points. In information theory terms, it provides a representation of Shannon’s work on lattice [116] further developed by Han [56]. $H_{k}$ and $I_{k}$, as real continuous functions, provide a ranking of the lattices at each dimension $k$. It is the ranking, i.e. the relative values of information, which matters and comes out of the homological approach, rather than the absolute values. The principle of $H_{k}$ and $I_{k}$ landscapes is illustrated in Figure 3 for $n=4$. $H_{k}$ and $I_{k}$ analyse quantify the variability-randomness and statistical dependences at all dimensions $k$, respectively, from $1$ to $n$, $n$ being the total number of variables under study. The $H_{k}$ landscape represents the values of joint entropy for all k-tuples of variables as a function of the dimensions $k$, the number of variables in the k-tuple, together with the associated edges-paths of the lattice (in grey). The $I_{k}$ landscape represents the values of mutual-information for all k-tuples of variables as a function of the dimension $k$, which is the number of variables in the $k$-tuple. Figure 3 gives two theoretical extremal examples of such landscapes, one for independent identically distributed variables (totally disordered) and one for fully dependent identically distributed variables (totally ordered). The degeneracy of $H_{k}$ and $I_{k}$ values is given by the binomial coefficient (color code in Figure 3), hence allowing to derive the normal exact expression of the information landscapes in the assymptotic infinite dimensional limit ($n\rightarrow\infty$) by application of Laplace-Lemoivre theorem. These are theoretical extremal examples: $H_{k}$ and $I_{k}$ landscapes effectively computed and estimated on biological data with finite sample are shown in the paper [16, 17, 125], and in practice the finite sample size ($m$) may impose some bounds on the landscapes.

3.4 Information Paths and Minimum Free Energy Complex

In this section we establish that information landscapes and paths encode directly the basic equalities, inequalities and functions of information theory and allow us to obtain the minimum free energy complex that we estimate on data.

3.4.1 Information Paths (definition)

On the discrete simplicial information lattice $\Delta_{k}$, we define a path of degree $k$ as a sequence of edges of the lattice that begins at the least element of the lattice (the identity-constant ”0”), travels along edges from vertex to vertex of increasing dimension and ends at the greatest element of the lattice of dimension $k$. Information paths are defined on both joint-entropy and meet-mutual information semi-lattices, and the usual joint-entropy and mutual-information functions are defined on each element of such paths. Entropy path and information path of degree $k$ are noted $HP_{k}$ and $IP_{k}$, respectively, and the set of all information paths is noted $\mathcal{HP}_{k}=\{HP_{i}\}_{i\in{1,...,k!}}$ for the entropy paths, and $\mathcal{IP}_{k}=\{IP_{i}\}_{i\in{1,...,k!}}$ for the mutual-information paths. We have the theorem:

Theorem 3.2**.**

The two sets of all information paths $\mathcal{HP}_{k}$ and $\mathcal{IP}_{k}$ in the simplicial information structure $\Delta_{k}$ are both in bijection with the symmetric group $S_{k}$. Notably, there are $k!$ information paths in $\Delta_{k}$.

Proof: by simple enumeration, an edge of dimension $m$ connects $k-m$ edges of dimension $m+1$, the number of paths is hence $(k-0).(k-1)....(k-k+2).(k-k+1)=k!$, hence the conclusion $\square$.

A given path can be identified with a permutation or a total order by extracting the missing variable in a previous node when increasing the dimension, for example the mutual-information path in $\Delta_{4}$: $IP_{i}=0\rightarrow(0,X_{2})\rightarrow(0,X_{1},X_{2})\rightarrow(X_{1},X_{2},X_{4})\rightarrow(0,X_{1},X_{2},X_{3},X_{4})$ can be noted as the permutation $\sigma$:

[TABLE]

We note an information path with arrows, giving for the previous example $IP_{i}=(0\rightarrow X_{2}\rightarrow X_{1}\rightarrow X_{4}\rightarrow X_{3})$. These paths shall be seen as the automorphisms of $\{1,2.....k\}=[k]$ and the space of entropy and mutual information paths can be endowed with the structure of two opposite symmetric groups $S_{k}$ and $S_{k}^{opp}$. The equivalence of the set of paths and symmetric group only holds for the subcase of simplicial structures, and the information paths in the lattice of partition are obviously much richer. More precisely, the subset of simplicial information paths in the lattice of partitions corresponds to the automorphisms of the lattice. It is known that the finite symmetric group is the automorphism group of the finite partition lattice [23]. The geometrical realization of information paths $\mathcal{IP}_{k}$ and $\mathcal{HP}_{k}$ consists in two dual permutohedron (see Postnikov [108]), and gives the informational version of the work of Matúš on conditional probability and permutohedron [90].

3.4.2 Derivatives, inequalities and conditional mutual information negativity

Derivatives of information paths:

In the information landscapes, the paths $HP_{i}$ and $IP_{i}$ are piecewise linear functions $IP_{i}(k)$ with $IP_{i}(k)=I_{k}$ where $I_{k}$ is the mutual-information of the k-tuple of variables pertaining to the path $IP_{i}$. We define the first derivatives of the paths for both entropy and mutual information structures as piecewise linear functions:

First derivative of entropy path: the first derivative of an entropy path $HP_{i}(k)$ is the conditional information $(X_{1},...,X_{k-1}).H(X_{k};\mathbb{P})$:

[TABLE]

This derivative is illustrated in the graph of Figure 4a. It implements the chain rule of entropy $H_{k+1}-H_{k}=(X_{1};...;\widehat{X_{i}};...;X_{k+1}).H(X_{i})$ (equation 12), and in homology provides a diagram where conditional entropy is a simplicial coface map $(X_{1};...;\widehat{X_{i}};...;X_{k+1}).H(X_{i})=d^{i}:~{}X^{k}\rightarrow X^{k+1}$, as a simplicial special case of Hochschild coboundaries 2.3.

First derivative of mutual information path: the first derivative of an information path $IP_{i}(k)$ is minus the conditional information $(X_{k}).I(X_{1},...,X_{k-1};\mathbb{P})$:

[TABLE]

This derivative is illustrated in the graph of Figure 4b. It implements the chain rule of mutual-information $I_{k-1}-I_{k}=X_{k}.I_{k-1}$ (equation 13), and in homology provides a diagram where minus the conditional mutual-information is a simplicial coface map $X_{i}.I(X_{1};...;\widehat{X_{i}};...;X_{k+1})=d^{i}:~{}X^{k}\rightarrow X^{k+1}$, introduced in section 2.3.

Bounds of the derivatives and information inequalities

The slope of entropy paths is bounded by the usual conditional entropy bounds ([147] p.27-28). Its minimum is [math] and is achieved in the case where $X_{k+1}$ is a deterministic function of ($X_{1},...,X_{k}$) (lower dashed red line in Figure 4a). Its global upperbound is $\max H_{k+1}=k.\ln(N_{1}...N_{k+1})$ and its sharp bound given by $(X_{1};...;\widehat{X_{i}};...;X_{k+1}).H(X_{i})\leq H(X_{i})$ is achieved in the case where $X_{k+1}$ is independent of $X_{1},...,X_{k}$ (we have $H_{k+1}=H_{k}+H(X_{k+1})$ (higher dashed red line in Figure 4a). Hence, any entropy path lies in the (convex) entropy cone defined by the 3 points labeled $H_{k}$, $\min H_{k+1}$ and $\max H_{k+1}$: the 3 vertices of the cone depicted as a red surface in Figure 4a and called the Shannonian Cone following Yeung’s seminal work [148]. The behavior of a mutual-information path and the bounds of its slope are richer and more complex than the preceding conditional entropy:

• •

For $k=2$, the conditional information is the conditional entropy $X_{i}.I(X_{j})=X_{i}.H(X_{j})$ and has the same usual bounds $0\leq X_{i}.I(X_{j})\leq I(X_{j})$.

• •

For $k=3$ the conditional mutual-information $X_{i}.I(X_{j};X_{h})$ is always positive or null $X_{i}.I(X_{j};X_{h})\geq 0$ and hence $I_{2}\geq I_{3}$ ([147] p.26, the opposite of th.2.40 p.30), whereas the higher limit is given by $X_{i}.I(X_{j};X_{h})\geq\min\left(X_{i}.H(X_{j}),X_{i}.H(X_{h})\right)$ ([89] th.2.17), with equality iff $X_{j}$ and $X_{h}$ are conditionally independent given $X_{i}$, and implying that the slope from $k=2$ to $k=3$ increases in the $I_{k}$ landscape.

• •

For $k>3$, $X_{k}.I(X_{1};..;X_{k-1})$ can be negative as a consequence of the preceding inequalities. In terms of information landscape this negativity means that the slope is positive, hence that the information path has crossed a critical point, a minimum. As expressed by theorem 2.2, $X_{k}.I(X_{1};..;X_{k-1})<0$ iff $I_{k}<I_{k+1}$. The minima correspond to zeros of conditional information (conditional independence) and hence detect cocycles in the data. The results on information inequalities define as ”Shannonian” [145, 149, 91] the set of inequalities that are obtained from conditional information positivity ($X_{i}.I(X_{j};X_{h})\geq 0$) by linear combination, which forms a convex ”positive” cone after closure. ”Non-Shannonian” inequalities could also be exhibited [145][149], hence defining a new convex cone that includes and is strictly larger than the Shannonian set. Following Yeung’s nomenclature and to underline the relation with his work, we call the positive conditional mutual-information cone (surface colored in red in Figure 4b) the ”Shannonian” cone and the negative conditional mutual-information cone (surface colored in blue in figure 4b) the ”non-Shannonian” cone.

3.4.3 Information paths are random processes: topological 2nd law of thermodynamic and entropy rate

Here we present the dynamical aspects of information structures. Information paths provide directly the standard definition of a stochastic process and it imposes how the time arrow appears in the homological framework, how time series can be analyzed, how entropy rates can be defined (etc.).

Random (stochastic) process (definition [124]): A random process $\{X_{t},t\in T\}$ is a collection of random variables on the same probability space $(\Omega,\mathcal{B},P)$ and the index set $T$ is a totally ordered set.

A stochastic process is a collection of random variables indexed by time, the probabilistic version of a time series. Considering each symbol of a time series as a random variable, the definition of a random-stochastic process corresponds to the unique information paths $HP_{i}$ and $IP_{i}$ which total order is the time order of the series. We have the following lemma:

Lemma 1**.**

(Stochastic process and information paths): Let $(\Omega,\Delta^{k},P)$ be a simplicial information structure, then the set of entropy paths $\mathcal{HP}_{k}$ and of mutual-information paths $\mathcal{IP}_{k}$ are in one to one correspondence with the set of stochastic processes $\{X_{t},t\in T,|T|=k\}$.

Proof: direct from the definitions $\Box$.

As we previously stated, these paths are also automorphisms of $\{1,2.....k\}=[k]$. We obtain immediately the topological version of the second law of thermodynamic, which improves the result of Cover [35]:

Theorem 3.3**.**

(Stochastic process and information paths): Let $(\Omega,\Delta^{k},P)$ be a simplicial information structure, then the entropy of a stochastic process can only increase with time.

Proof: given the correspondence we just established, the statement is equivalent to $H(X_{1},...,X_{k})\geq H(X_{1},...,X_{k-1})$, which is a direct consequence of conditional entropy positivity and the chain rule of information with $k=-1/\ln 2$. The generalization with respect to the stationary Markov condition used by Cover comes from the remark that in any case the indexing set of the variable is a total order. Note that the homological formalism imposes an ”initial” minimally low entropy state $H(0)=I(0)=0$ (a usual assumption in physics), the constant and zero degree homology, which has to have at least 1 component to talk about the cohomology $\Box$.

Remark: the meaning of this theorem in common terms was summarized by Gabor and Brillouin: ”you can’t have something for nothing, not even an observation” [26]. This increase in entropy is illustrated in Figure a. The usual stochastic approach of time series assumes a Markov chain structure, imposing peculiar statistical dependences that restrict memory effects (cf. the informational characterization of Markov chains in the associated paper [16]). The consideration of stochastic processes without restriction allows any kind of dependences and arbitrary long historical and ”non trivial” memory. From the biological point of view it formalizes the phenomenon of arbitrary long-lasting memory. From the physical point of view, without proof, such a framework appears as a classical analog of the consistent or decoherent histories developed notably by Griffiths [55], Omnes [103], and Gell-Mann and Hartle [50]. The information structures impose a stronger constraint of a totally ordered set (or more generally a weak ordering) than the preorder imposed by Lieb and Yngvason [79] to derive the second law. It is also interesting to note that even in this classical probability framework, the entropy cone (the topological cone depicted in Figure 4a) imposed by information inequalities, when considered with this time ordering, is a time-like cone (much-like the special relativity cone), but with the arguably remarkable fact that we did not introduce any metric.

The stochastic process definition allows to define the finite and asymptotic information rate: the finite information rate $r$ of an information path $HP_{i}$ is $r=\frac{H_{k}}{k}$. The asymptotic information rate $r$ of an information path $HP_{i}$ is $r=\lim_{k\rightarrow\infty}\frac{H_{k}}{k}$. It requires the generalization of the present formalism to the infinite dimensional setting or infinite information structures, which is not trivial and will be investigated in further work.

3.4.4 Local minima and critical dimension

The derivative of information paths allows to establish the lemma on which is based the information path analysis. A critical point is said to be non-trivial if at this point the sign of the derivative of the path, i.e. the conditional information, changes.

Lemma 2**.**

(local minima of information paths): if $X_{k}.I(X_{1};..;X_{k-1})<0$ then all paths from [math] to $I_{k}$ passing by $I_{k-1}$ have at least one local minimum. In order for an information path to have a non-trivial critical point, it is necessary that $k>3$, the smallest possible dimension of a critical point being $k=3$.

Proof: it is a direct consequence of the definitions of paths and of conditional 2-mutual-information $X_{k}.I_{2}$ positivity ($X_{k}.I_{2}\geq 0$, cf. theorem • ‣ 3.4.2)$\Box$.

Note that by definition a local minimum can be a global minimum. We will call, if it exists, the dimension $k$ of the first local minimum of an information path the first informational critical dimension of the information path $IP_{i}$, and note it $k_{i_{1}}$. This allows us to define maximal information paths:

Positive information path (definition): A positive information path is an information path from [math] to a given $I_{k}$ corresponding to a given $k$-tuple of variables such that $I_{k}<I_{k-1}<...<I_{1}$.

Maximal Positive information path (definition): A maximal positive information path is a positive information path of maximal length. More formally, a maximal positive information path is a positive information path that is not a proper subset of positive information paths.

The definitions make coincide positive information paths and maximal positive information path with chains (faces) and maximal chains (facets), respectively. The maximal positive information path stops at the first local minimum of an information path, if it exists. The first informational critical dimension $k_{i_{1}}$ of a time series $IP_{i}$, whenever it exists, gives a quantification of the duration of the memory of the system.

3.4.5 Sum over paths and mean information path

As previously, for $k=1$, $IP_{i}(1)$ can be identified with the self-internal energy and for $k\geq 2$, $IP_{i}(k)$ corresponds to the k-free-energy of a single path $IP_{i}$. The chain rule of mutual information (equation 15) and the derivative of an $IP_{i}$ path (equation 37) implies that the k-free-energy can be obtained from a single path:

[TABLE]

Hence, the global thermodynamical relation 25 can be understood as the sum over all paths, the sum over informational histories: the classical, discrete and informational version of the path integrals in statistical physics [46]. Indeed considering an inverse relation between time and dimension $t=\frac{1}{n}$ in the probability expression 2 for iid processes gives the usual expression of a unitary evolution operator $p_{\bullet\bullet...k...\bullet}=e^{\frac{t.E_{ij...n}}{k}}$. Free-information-energy integrates over the simplicial structure of the whole lattice of partitions over degrees $k\geq 2$, which further justifies its free energy name.

In order to obtain a single state function instead of a group of $k!$ paths-functions, we can compute the mean behavior of the information structure, which is achieved by defining the mean $H_{k}$ and $I_{k}$, noted $\langle H_{k}\rangle$ and $\langle I_{k}\rangle$:

[TABLE]

and

[TABLE]

For example, considering $n=3$, then $\langle I_{2}\rangle=\frac{I(X_{1};X_{2})+I(X_{1};X_{3})+I(X_{2};X_{3})}{3}$. This defines the mean mutual information path and a mean entropy path noted $\langle HP\rangle(k)$ and $\langle IP\rangle(k)$ in the information landscape. The case $k=2$ of those functions introduced in [17], is studied in Merkh and Montúfar [94] with a characterization of the degeneracy of their maxima and are called Factorized Mutual-Information. As previously, $\langle IP\rangle(1)$ can be identified with the mean self-internal energy $U(X^{n}_{hom};P_{N})$ and for $k>1$ $\langle IP\rangle(k)$ to the mean k-free-information-energy $G(X^{n}_{hom};P_{N})$, giving the usual isotherm relation:

[TABLE]

The computation of the mean paths corresponds to an idealized information structure $X^{n}_{hom}$ for which all the variables would be identically distributed, would have the same entropy, and would share the same mutual information $I_{k}$ at each dimension $k$: a homogeneous information structure, with homogeneous high-dimension k-body interactions. Like usually achieved in physics notably in mean-field theory (for example Weiss [140] or Hartree), it aims to provide a single function summarizing the average behavior of the system (we will see that in practice it misses the important biological structures, pointing out the constitutive heterogeneity of biological systems see 4.1). Using the same dataset and results presented in [16, 17, 125], the $\langle IP\rangle(k)$ paths estimated on genetic expression data set are shown for two population A and population B of neurons in Figure 5. We quantified the gene expression levels for 41 genes in two populations of cells (A or B) as presented in [16, 17, 125]. We estimated $H_{k}$ and $I_{k}$ landscapes for these two populations and for two sets of genes (”genes of interest” and ”non relevant”) according to the computational and estimation methods presented in [16, 17, 125]. The available computational power restricts the analysis to a maximum of $n=21$ variables (or 21 dimensions), and imposed us to divide the genes between the two classes ”genes of interest” and ”non relevant”. The 21 genes of interest were selected within the 41 quantified genes according to their known specific involvement in the function of population A cells.

Figure 5 exhibits the critical phenomenon usually encountered in condensed matter physics, like the example of Van-der-Walls interactions [106]. Like any $I_{k}$ path, $\langle IP\rangle(k)$ can have a first minimum with a critical dimension $k_{i_{1}}$ that could be called the homogeneous critical dimension. For the 21 genes of interest (whose expression levels, given the literature, are expected to be linked in these cell types) the $\langle I_{k}\rangle$ path exhibits a clear minimum at the critical dimension $k_{i_{1}}=4$ for population A neurons and $k_{i_{1}}=5$ population B neurons, reproducing the usual free-energy potential in the condensed phase for which n-body interactions are non-negligible. For the 20 other genes, less expected to be related in these cell types, the $\langle I_{k}\rangle$ path exhibits a monotonic decrease without a non-trivial minimum, which corresponds to the usual free-energy potential in the uncondensed-disordered phase for which the n-body interactions are negligible. Indeed, as shown in the work of Xie and colleagues [144], the tensor network renormalization approach of n-body interacting quantum systems gives rise to an expression of the free-energy as a function of the dimension of the interactions, in the same way than achieved here.

3.4.6 Minimum free energy complex

The analysis of information paths that we now propose aims to determine all the first critical points of information paths, in other words to determine all the information paths for which conditional information stays positive, and all first local minima of the information landscape, that can also be interpreted as a conditional independence criterion. Such an exhaustive characterization would give a good description of the landscape and of the complexity of the measured system. The qualitative reason for considering only the first extrema for the data analysis is that, beyond that point, mutual information diverges (as section 3.4.4 explains) and the maximal positive information paths correspond to stable functional modules in the application to data (gene expression).

A more mathematical justification is that they define the facets of a complex in our simplicial structure, which we will call the minimum energy complex of our information structure, underlining that this complex is the formalization of the minimum free energy principle in a degenerate case.

We now obtain the theorem that our information path analysis aims to characterize empirically:

Theorem 3.4**.**

(Minimum free energy complex): the set of all maximal positive information paths forms a simplicial complex that we call the minimum free energy complex. Moreover, the dimension-degree of the minimum free energy complex is the maximum of all the first informational critical dimensions ($d=\max k_{i_{1}}$), if it exists, or the dimension of the whole simplicial structure $n$. The minimum free energy complex is noted $X^{+d}$. A necessary condition for this complex not to be a simplex is that its dimension is greater or equal to four ($d\geq 4$).

Proof: It is known that there is a one to one correspondence between simplicial complexes and their set of maximal chains (facets) (see [123] p.95 for example). The last part follows from Lemma 1. $\Box$.

In simple words, the maximal faces, e.g. the maximal positive information paths, encode all the structures of the minimum free energy complex. Figure 6 illustrates one of the simplest examples of a minimum free energy complex that is not a simplex, of dimension four in a five-dimensional simplicial structure of information $\Delta_{5}$.

We define the minimum free energy characteristic as:

[TABLE]

, where the component with dimension higher than one is a free energy. In the example of Figure 6 it gives:

[TABLE]

We propose that this complex defines a complex system:

Complex system (definition): A complex system is a minimum free energy complex.

It has the merit to provide a formal definition of complex systems as simple as the definition of an abstract simplicial complex can be, and to be quite consensual with respect to some of the approaches in this domain, as reviewed by Newman [98]. Notably, it provides a formal basis to define some of the important concepts in complex systems: emergence being the coboundary map, imergence the boundary map, synergy being information negativity, organization scales being the ranks of random-variable lattices, a collective interaction being a local minimum of free-energy, diversity being the multiplicity of these minima quantified by the number of facets, a network being a 1-complex, a network of network being a 1-complex in hyper-cohomology.

The interpretation in terms of sum over paths in the complex is direct as it sums over paths until conditional independence. We called it the minimum free energy complex but could have called it instead the positive or instantaneous complex because its facets appear as the boundaries of the ”present” structure, but it obviously contains all the past-history and the memory of the structure (notably encoded in the negative $I_{k}$ that are necessarily non-Markovian). The topological formalization of the minimum energy allows the coexistence of numerous local minima, a situation usually encountered in complex systems (slow aging) such as frustrated glasses and K-sat problems [95, 96] which settings correspond here to the case of $n$ binary random variables, $N_{1}=...=N_{2}=2$. The existence of the frustration effect, due to the multiplicity of these local minima in the free energy landscape [132], has also been one of the main difficulties of the condensed matter theory. Matsuda could show that $I_{k}$ negativity is a signature of frustration [89]. The first axioms of DFT consider that probability densities of $n^{\prime}$ elementary bodies are each in a 3-dimensional space [58, 70], defining a whole simplicial structure of dimension $n=3n^{\prime}$, commonly called the configuration space. When considered with the physical axiom of a configuration space, the theorem 3.4 implies that, while the minimum free information energy complex of an elementary body can only be a simplex, the configuration space of $n^{\prime}$ elementary bodies can be a complex with (quite) arbitrary topology. In simple terms, this settles the elementary components of the configuration space as 3-simplices, which composition can give arbitrarily complicated k-complexes. This idea is in resonance with the triangulations of space-time that arose notably from the work of Wheeler [141] and Penrose [107], like spin foams [113] and causal sets [118], while we only considered here classical probabilities.

4 Discussion

4.1 Complexity through finite dimension non-extensivity and non iid

Statistical physics without statistical limit?

The measure of entropy and information rate on data (the evolution of entropy $H_{k}$ when the number of variables $k$ increases) has a long history. Originally, in the work of Strong and colleagues [120], and as usual in information theory and statistical physics, it was considered that the ”true” entropy was given in the asymptotic limit $\lim_{n\rightarrow\infty}H_{n}$ under stationarity or stronger assumptions. As explained in section 2.1 (see also the note of Kontsevitch [72], in the work of Baez, Fritz and Leinster [9]), and extensively in the statistical physic works of Niven [101, 100, 99], entropy does not need asymptotic or infinite assumptions such as Stirling approximation to be derived. Here and in the associated paper [16], we tried to understand, explore, and exploit this observation. Rather than being interested in the asymptotic limit (the infinite dimensional case) and absolute values of information, the present analysis focuses on the finite version of the ”slow approach of the entropy to its extensive asymptotic limit” that Grassberger [54], and then Bialek, Nemenman and Tishby proposed to be ”a sign of complexity” [22], ”complexity through non-extensivity” (see also Tsallis [131]). In short, we consider the non-extensivity of information before considering its asymptotic limit. Considering a statistical physics without statistical limit could be pertinent for the study of ”small” systems, which concerns biological systems. Their small size allows them to harness thermal fluctuations, and impose their investigation with out-of equilibrium methods, as exposed in the work of Ritort and colleagues, reviewed in [112]. The $H_{k}$ and $I_{k}$ landscapes presented here give a detailed expression of the ”signs of complexity” and non-extensivity, for such small size systems (finite dimension $k$), and give a finite dimensional geometric view of the ”slow approach of the entropy to its extensive asymptotic limit”. In a sense, what replaces here the large number limits, Avogadro number consideration (etc.), is the combinatorial explosions of the different possible interactions: in the same way as in Van Der Walls paradigm, a combinatorial number of weak interactions can lead to a strong global interaction. In practice, like for any empirical investigation, the finite dimensional case imposes restrictions on the conclusions, and basically reminds us that we have not measured everything. Some relevant random variables for the observed system may be absent from the analysis, and adding such variables could reveal new interactions that are effectively constitutive and relevant to the system. Among all possible structures of data, one is universal: data and empirical measures are discrete and finite, as emphasized by Born [24], and fully justify the cohomological approach used here originating in topos (designed by Grothendieck to hold the discrete and continuous in a single hand), which was originally constructed to handle in a common framework the Lie and Galois theory, continuous and discrete symmetries.

Naive estimations let the data speaks, lack of iid or mean-field assumptions let differentiate the objects

One of the striking results of the data analysis as presented here and in the associated paper [16] concerns the relatively low sample size ($m=41$ and $m=111$ for the analysis with cells as variable and with genes as variables respectively) required to obtain satisfying results in relatively high dimensions ($k=10$ and $k=6$ respectively). Satisfying results means here that they predict already known results reported in the biological literature, or in agreement with experts labels. In [87], Nemenman and colleagues, who developed the problematic of the sampling problem, state in the introduction ”entropy may be estimated reliably even when inferences about details of the underlying probability distribution are impossible. Thus the direct estimation of dependencies has a chance even for undersampled problems” and conclude that ”a major advantage of our definition of statistical dependencies in terms of the MaxEnt approximations is that it can be applied even when the underlying distributions are undersampled”. The present analysis agrees and confirms their conclusion. The method applied here is quite elementary. It does not make assumptions of an expected or true distribution, of maximum entropy distribution or pairwise interaction Hamiltonian, coupling constant or metric, of stationarity or ergodicity or iid process, Markov chain, or underlying network structure (…) or whatever prior that would speak in place of the data. It just considers numerical empirical probabilities as expressed by Kolmogorov axioms ([71] chap.1), which he called the ”generalized fields of probability” because it does not assume the 6th axiom of continuity. Rather than fixing a model with priors, the present formalism allows the raw data to impose freely their specific structure to the model, what is usually called the naive approach or naive estimation. If one accepts that a frequentist theory and interpretation of probability is mathematically valid ([71], chap.1), one may then conclude that a frequentist theory of entropy and information may also hold, and moreover directly fulfills the usual requirement of observability in theoretical physics recalled by Born in his Nobel lecture [24]. This frequentist elementary consideration is not trivial mathematically notably when considered from the number theoretic point of view. For example, the combinatoric of integer partitions of $m$ could be investigated in the general information structure (partition) context, which up to our knowledge has not been achieved in the context of probability and information. Up to our knowledge, all previous studies that tried to quantify statistical dependences by information methods (with more than 3 variables) used total correlation [129, 87], and crucially assumed that the interaction between the variables are homogeneous which corresponds to usual mean field assumption, and to the iid case of mean information 3.4.5 presented here. The combinatorial decomposition proposed allows to identify heterogeneous classes within the set of variables which would not have been possible using homogeneous assumptions. We hence believe that such combinatorial approach will play a key role in future machine learning developments and automatic classification problems. Notably, we only explored here the smallest and simplest combinatorics arising in homology, and Vigneaux already identified q-multinomial extensions of this combinatorics associated with Tsallis entropies [134]. Notably, as stressed in introduction, those combinatorial decomposition can be understood as providing a geometrically constrained architecture to neural networks generalization.

4.2 Epigenetic topological learning - biological diversity

In place of the MaxEnt principle, we proposed an almost synonymous least energy principle equivalent here to a homological complex (finite and without metric assumptions). Mathematically, we took profit of the fact that whether the maximum of entropy functional is always unique and in a sense normative, the minima of $I_{k}$ functionals exhibit a rich structure of degeneracy, generated by the ”non-Shannonian set” [145, 149, 91] and conjectured to be at least as rich as topological links can be. We proposed that this multiplicity of minima accounts for biological diversity, or more precisely that the number of facets of this complex quantifies the diversity in the system. The application to cell type identification presented in the associated paper [16] gives a preliminary validation of such quantification. Moreover, the definition of a complex system as the minimum free-energy complex given in section 44, underlining that diversity is just the multiplicity of the minima, is in agreement with Waddington’s original work [137] (see Figure 7b). In the allegory of Waddington’s epigenetic landscapes, whatever the ball, it will always fall down, a statement that can be assimilated to the second law of thermodynamic. But doing so, it will be able to take different paths: diversity comes from the multiple minima. The explanation by Waddington of such landscape is a ”complex system of interactions” that can be formalized by the minimum free energy complex with interactions corresponding to the $I_{k}$. Moreover, formalisms assuming that the variables are identically distributed, as for the homogeneous systems described in the section on mean paths 3.4.5, will display a single first minima (one facet, a simplex), and hence no diversity. Sharing the same aims, Teschendorff and Enver, and then Jin and colleagues, proposed an alternative interpretation of Waddington’s landscape in terms of signaling entropy [127] and of probability transitions [65], respectively.

Following Thom’s topological morphogenetic view of Waddington’s work [128], we propose that $I_{k}$ landscape, paths and minimum free energy complex provide a possible informational formalization of Waddington’s epigenetic complex landscape and cell fates (cf. Figure 7). This formalization of Waddington’s epigenetic view is consistent with the machine learning formalization of Hebbian epigenetic plasticity. From the pure formal view, the models of Hebbian neural learning like Hopfield’s network, Boltzmann machines, the Infomax models proposed by Linsker, Nadal and Parga, Bell and Sejnowski [80, 97, 19]) can be viewed as binary variables subcases of a generic N-ary variable epigenetic developmental process. For example, Potts model were implemented for the simulation of cell-based morphogenesis by Glazier and colleagues [31]. Hence the topological approach can allow the treatment of neural learning and development on the ground of a common epigenetic formalism, in agreement with biological results pointing out the continuum and ”entanglement” of the biological processes underlying development and learning [49]. In terms of the current problematics of neuroscience, such generalization allows on a formal level to consider an analog coding in place of a digital coding, and the methods developed here can be applied to studies investigating (discrete) analog coding.

Moreover, following all the work of these last decades on the application of statistical physics to biological systems (some of them cited in this article), we propose that the epigenetic process implements the first two laws of thermodynamics, which weak topological versions are proposed to hold in the raw data space (without phase space or symplectic structure, cf. section 3.4.3). As previously underlined, the condition for such an inscription of living organism dynamic into classical statistical physics to be legitimate is that the considered variables correspond to phase space variables.

Previous versions

Previous Version: a partial version of this work has been deposited in the method section of Bioarxiv 168740 in July 2017 and preprints [17].

Acknowledgement

This work was funded by the European Research Council (ERC consolidator grant 616827 CanaloHmics to J.M.Goaillard) and Median Technologies, developed at Median Technologies and UNIS Inserm 1072 - Université Aix-Marseille, and at Institut de Mathématiques de Jussieu - Paris Rive Gauche (IMJ-PRG), and thanks previously to supports and hostings since 2007 of Max Planck Institute for Mathematic in the Sciences (MPI-MIS) and Complex System Instititute Paris-Ile-de-France (ISC-PIF). This work addresses a deep and warm acknowledgement to the researchers who helped its realization: D.Bennequin, J.M.Goaillard, Hong Van le, G.Marrelec, M.Tapia and J.P. Vigneaux; or supported-encouraged it: H.Atlan, F.Barbaresco, H.Bénali, P.Bourgine, F.Chavane, J.Jost, A.Mohammad-Djafari, JP.Nadal, J.Petitot, A.Sarti, J.Touboul. A partial version of this work has been deposited in the method section of Bioarxiv 168740 in July 2017 and preprints [17].

Abbreviations

The following abbreviations are used in this manuscript:

[TABLE]