Infinite-dimensional Folded-in-time Deep Neural Networks

TL;DR

This paper extends a recently proposed folded-in-time neural network model to an infinite-dimensional setting, enabling more rigorous analysis, flexible weight functions, and gradient-based training, including recurrent neural networks.

Contribution

It introduces an infinite-dimensional generalization of folded-in-time neural networks with Lebesgue integrable weights and a functional back-propagation algorithm for training.

Findings

Provides a mathematically rigorous framework for folded-in-time neural networks.

Enables gradient descent training of infinite-dimensional weights.

Realizes recurrent neural networks with minimal modifications.

Abstract

The method recently introduced in arXiv:2011.10115 realizes a deep neural network with just a single nonlinear element and delayed feedback. It is applicable for the description of physically implemented neural networks. In this work, we present an infinite-dimensional generalization, which allows for a more rigorous mathematical analysis and a higher flexibility in choosing the weight functions. Precisely speaking, the weights are described by Lebesgue integrable functions instead of step functions. We also provide a functional back-propagation algorithm, which enables gradient descent training of the weights. In addition, with a slight modification, our concept realizes recurrent neural networks.