Information erasure lurking behind measures of complexity
Karoline Wiesner, Mile Gu, Elisabeth Rieper, and Vlatko Vedral

TL;DR
This paper reveals that the difference between two key complexity measures in systems is due to information erasure during forecasting, proposing an efficiency bound that guides the development of better models with minimal information loss.
Contribution
It establishes a quantitative link between statistical complexity and excess entropy, interpreting their difference as information erasure and defining an efficiency bound for models.
Findings
The difference between the measures quantifies information erasure.
An efficiency bound analogous to thermodynamics is derived.
Good models are those with minimal information erasure.
Abstract
Complex systems are found in most branches of science. It is still argued how to best quantify their complexity and to what end. One prominent measure of complexity (the statistical complexity) has an operational meaning in terms of the amount of resources needed to forecasting a system's behaviour. Another one (the effective measure complexity, aka excess entropy) is a measure of mutual information stored in the system proper. We show that for any given system the two measures differ by the amount of information erased during forecasting. We interpret the difference as inefficiency of a given model. We find a bound to the ratio of the two measures defined as information-processing efficiency, in analogy to the second law of thermodynamics. This new link between two prominent measures of complexity provides a quantitative criterion for good models of complex systems, namely those with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsStatistical Mechanics and Entropy · Advanced Thermodynamics and Statistical Mechanics · Complex Systems and Time Series Analysis
