On sampling and modeling complex systems

TL;DR

This paper explores how limited sampling and partial knowledge affect modeling complex systems, revealing that the distribution of known variables follows a Boltzmann form and identifying conditions for predictability and informativeness of samples.

Contribution

It introduces a generic framework linking sampling limitations to system modeling, deriving the Boltzmann distribution form, and characterizing sample informativeness and variable relevance.

Findings

Distribution of known variables follows Boltzmann form.

Predictability is limited when relevant variables exceed a threshold.

Zipf's law emerges at the crossover between under-sampling and sufficient sampling.

Abstract

The study of complex systems is limited by the fact that only few variables are accessible for modeling and sampling, which are not necessarily the most relevant ones to explain the systems behavior. In addition, empirical data typically under sample the space of possible states. We study a generic framework where a complex system is seen as a system of many interacting degrees of freedom, which are known only in part, that optimize a given function. We show that the underlying distribution with respect to the known variables has the Boltzmann form, with a temperature that depends on the number of unknown variables. In particular, when the unknown part of the objective function decays faster than exponential, the temperature decreases as the number of variables increases. We show in the representative case of the Gaussian distribution, that models are predictable only when the number of relevant variables is less than a critical threshold. As a further consequence, we show that the information that a sample contains on the behavior of the system is quantified by the entropy of the frequency with which different states occur. This allows us to characterize the properties of maximally informative samples: in the under-sampling regime, the most informative frequency size distributions have power law behavior and Zipf's law emerges at the crossover between the under sampled regime and the regime where the sample contains enough statistics to make inference on the behavior of the system. These ideas are illustrated in some applications, showing that they can be used to identify relevant variables or to select most informative representations of data, e.g. in data clustering.