Arbitrage equilibrium and the emergence of universal microstructure in deep neural networks

TL;DR

This paper introduces a game-theoretic framework called statistical teleodynamics to explain the microstructure of deep neural networks, revealing universal properties like arbitrage equilibrium and lognormal weight distributions supported by empirical data.

Contribution

It presents a novel game-theoretic approach to understanding neural microstructure, identifying arbitrage equilibrium and universal distributions as key features.

Findings

Neural microstructure exhibits arbitrage equilibrium with equal effective utility.

Connection weights and outputs follow lognormal distributions in trained networks.

Empirical data from seven large-scale networks support the theoretical predictions.

Abstract

Despite the stunning progress recently in large-scale deep neural network applications, our understanding of their microstructure, 'energy' functions, and optimal design remains incomplete. Here, we present a new game-theoretic framework, called statistical teleodynamics, that reveals important insights into these key properties. The optimally robust design of such networks inherently involves computational benefit-cost trade-offs that are not adequately captured by physics-inspired models. These trade-offs occur as neurons and connections compete to increase their effective utilities under resource constraints during training. In a fully trained network, this results in a state of arbitrage equilibrium, where all neurons in a given layer have the same effective utility, and all connections to a given layer have the same effective utility. The equilibrium is characterized by the emergence of two lognormal distributions of connection weights and neuronal output as the universal microstructure of large deep neural networks. We call such a network the Jaynes Machine. Our theoretical predictions are shown to be supported by empirical data from seven large-scale deep neural networks. We also show that the Hopfield network and the Boltzmann Machine are the same special case of the Jaynes Machine.