Nonparametric Maximum Entropy Estimation on Information Diagrams

TL;DR

This paper introduces an advanced nonparametric maximum entropy estimation method for both discrete and continuous variables, improving accuracy and computational efficiency in complex systems analysis.

Contribution

It extends previous techniques to continuous variables, broadening the scope and applicability of maximum entropy estimation in information theory.

Findings

Performs well in undersampled regimes where other methods fail

Less computationally expensive with increasing variable cardinality

Effectively estimates network connectivity in real-world brain data

Abstract

Maximum entropy estimation is of broad interest for inferring properties of systems across many different disciplines. In this work, we significantly extend a technique we previously introduced for estimating the maximum entropy of a set of random discrete variables when conditioning on bivariate mutual informations and univariate entropies. Specifically, we show how to apply the concept to continuous random variables and vastly expand the types of information-theoretic quantities one can condition on. This allows us to establish a number of significant advantages of our approach over existing ones. Not only does our method perform favorably in the undersampled regime, where existing methods fail, but it also can be dramatically less computationally expensive as the cardinality of the variables increases. In addition, we propose a nonparametric formulation of connected informations and give an illustrative example showing how this agrees with the existing parametric formulation in cases of interest. We further demonstrate the applicability and advantages of our method to real world systems for the case of resting-state human brain networks. Finally, we show how our method can be used to estimate the structural network connectivity between interacting units from observed activity and establish the advantages over other approaches for the case of phase oscillator networks as a generic example.