Constraint Breeds Generalization: Temporal Dynamics as an Inductive Bias

TL;DR

This paper argues that biological-like physical constraints shape neural dynamics into a beneficial inductive bias, enhancing generalization across various learning tasks by leveraging temporal dynamics and phase-space properties.

Contribution

It introduces the concept of dynamical constraints as a novel inductive bias, supported by phase-space analysis and extensive evaluations demonstrating improved generalization in neural networks.

Findings

Proper dissipative dynamics compress phase space and promote invariant features.

A critical transition regime maximizes generalization capabilities.

Temporal integration architectures outperform static ones in leveraging invariants.

Abstract

Conventional deep learning prioritizes unconstrained optimization, yet biological systems operate under strict metabolic constraints. We propose that these physical constraints shape dynamics to function not as limitations, but as a temporal inductive bias that breeds generalization. Through a phase-space analysis of signal propagation, we reveal a fundamental asymmetry: expansive dynamics amplify noise, whereas proper dissipative dynamics compress phase space that aligns with the network's spectral bias, compelling the abstraction of invariant features. This condition can be imposed externally via input encoding, or intrinsically through the network's own temporal dynamics. Both pathways require architectures capable of temporal integration and proper constraints to decode induced invariants, whereas static architectures fail to capitalize on temporal structure. Through comprehensive evaluations across supervised classification, unsupervised reconstruction, and zero-shot reinforcement learning, we demonstrate that a critical "transition" regime maximizes generalization capability. These findings establish dynamical constraints as a distinct class of inductive bias, suggesting that robust AI development requires not only scaling and removing limitations, but computationally mastering the temporal characteristics that naturally promote generalization.