Convergence and scaling of Boolean-weight optimization for hardware reservoirs

TL;DR

This paper derives scaling laws for coordinate descent optimization of reservoir neural networks, demonstrating exponential convergence and linear scaling with network size, validated by photonic reservoir experiments.

Contribution

It provides the first analytical derivation of convergence and scaling laws for coordinate descent in hardware reservoirs, guiding future optimization strategies.

Findings

Convergence is exponential in the number of iterations.

Scaling of convergence is linear with the number of neurons.

Experimental results match theoretical predictions.

Abstract

Hardware implementation of neural network are an essential step to implement next generation efficient and powerful artificial intelligence solutions. Besides the realization of a parallel, efficient and scalable hardware architecture, the optimization of the system's extremely large parameter space with sampling-efficient approaches is essential. Here, we analytically derive the scaling laws for highly efficient Coordinate Descent applied to optimizing the readout layer of a random recurrently connection neural network, a reservoir. We demonstrate that the convergence is exponential and scales linear with the network's number of neurons. Our results perfectly reproduce the convergence and scaling of a large-scale photonic reservoir implemented in a proof-of-concept experiment. Our work therefore provides a solid foundation for such optimization in hardware networks, and identifies future directions that are promising for optimizing convergence speed during learning leveraging measures of a neural network's amplitude statistics and the weight update rule.