Dataset-learning duality and emergent criticality

TL;DR

This paper explores the duality between dataset and learning dynamics in neural networks, revealing how criticality and power-law fluctuations can emerge from dataset properties and be influenced by activation and loss functions.

Contribution

It introduces a novel dataset-learning duality map and demonstrates how criticality can arise in neural networks from dataset characteristics, modifiable by activation and loss functions.

Findings

Criticality can emerge from non-critical datasets.

Power-law fluctuations are influenced by activation and loss functions.

Duality map links dataset properties to learning dynamics.

Abstract

In artificial neural networks, the activation dynamics of non-trainable variables is strongly coupled to the learning dynamics of trainable variables. During the activation pass, the boundary neurons (e.g., input neurons) are mapped to the bulk neurons (e.g., hidden neurons), and during the learning pass, both bulk and boundary neurons are mapped to changes in trainable variables (e.g., weights and biases). For example, in feed-forward neural networks, forward propagation is the activation pass and backward propagation is the learning pass. We show that a composition of the two maps establishes a duality map between a subspace of non-trainable boundary variables (e.g., dataset) and a tangent subspace of trainable variables (i.e., learning). In general, the dataset-learning duality is a complex non-linear map between high-dimensional spaces. We use duality to study the emergence of criticality, or the power-law distribution of fluctuations of the trainable variables, using a toy model at learning equilibrium. In particular, we show that criticality can emerge in the learning system even from the dataset in a non-critical state, and that the power-law distribution can be modified by changing either the activation function or the loss function.