Parametric machines: a fresh approach to architecture search
Pietro Vertechi, Mattia G. Bergomi

TL;DR
This paper introduces a novel topological and functional analysis framework for neural network architecture search, generalizing neural networks and differential equations, and demonstrates improved performance on small datasets.
Contribution
It develops a formal topological framework for neural architectures, combining simple machines into complex ones, and introduces kernel-inspired networks for better small-data performance.
Findings
Kernel-inspired networks outperform classical neural networks on small datasets
Finite- and infinite-depth machines generalize neural networks and neural ODEs
Framework enables systematic architecture and parameter optimization
Abstract
Using tools from topology and functional analysis, we provide a framework where artificial neural networks, and their architectures, can be formally described. We define the notion of machine in a general topological context and show how simple machines can be combined into more complex ones. We explore finite- and infinite-depth machines, which generalize neural networks and neural ordinary differential equations. Borrowing ideas from functional analysis and kernel methods, we build complete, normed, infinite-dimensional spaces of machines, and we discuss how to find optimal architectures and parameters -- within those spaces -- to solve a given computational problem. In our numerical experiments, these kernel-inspired networks can outperform classical neural networks when the training dataset is small.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsModel Reduction and Neural Networks · Neural Networks and Applications · Machine Learning in Materials Science
