"Efficient Complexity": a Constrained Optimization Approach to the Evolution of Natural Intelligence

TL;DR

This paper proposes a constrained optimization framework based on information entropy to model the evolution of natural intelligence, explaining neural network effectiveness through resource-efficient approximation strategies.

Contribution

It introduces a novel formalism linking information theory, thermodynamics, and biophysics to describe natural learning processes under resource constraints.

Findings

Natural intelligence evolution can be viewed as resource-constrained optimization.

Neural networks' effectiveness is explained by their approximation strategies within this framework.

The approach unifies biological and artificial intelligence principles through entropy maximization.

Abstract

A fundamental question in the conjunction of information theory, biophysics, bioinformatics and thermodynamics relates to the principles and processes that guide the development of natural intelligence in natural environments where information about external stimuli may not be available at prior. A novel approach in the description of the information processes of natural learning is proposed in the framework of constrained optimization, where the objective function represented by the information entropy of the internal states of the system with the states of the external environment is maximized under the natural constraints of memory, computing power, energy and other essential resources. The progress of natural intelligence can be interpreted in this framework as a strategy of approximation of the solutions of the optimization problem via a traversal over the extrema network of the objective function under the natural constraints that were examined and described. Non-trivial conclusions on the relationships between the complexity, variability and efficiency of the structure, or architecture of learning models made on the basis of the proposed formalism can explain the effectiveness of neural networks as collaborative groups of small intelligent units in biological and artificial intelligence.