Learn Like The Pro: Norms from Theory to Size Neural Computation

TL;DR

This paper introduces a theory-based method to determine neural network sizes using a Learnability metric derived from dynamical systems, enabling size estimation without training data.

Contribution

It proposes a novel, training-free approach to estimate neural network sizes based on dynamical system norms, bridging theory and neural network design.

Findings

Provides exact sizing for neural networks with multiplicative nodes.

Offers tight lower bounds for classical feed-forward networks.

The approach aligns well with simulated assessments.

Abstract

The optimal design of neural networks is a critical problem in many applications. Here, we investigate how dynamical systems with polynomial nonlinearities can inform the design of neural systems that seek to emulate them. We propose a Learnability metric and its associated features to quantify the near-equilibrium behavior of learning dynamics. Equating the Learnability of neural systems with equivalent parameter estimation metric of the reference system establishes bounds on network structure. In this way, norms from theory provide a good first guess for neural structure, which may then further adapt with data. The proposed approach neither requires training nor training data. It reveals exact sizing for a class of neural networks with multiplicative nodes that mimic continuous- or discrete-time polynomial dynamics. It also provides relatively tight lower size bounds for classical feed-forward networks that is consistent with simulated assessments.