Training Neural Networks Using Reproducing Kernel Space Interpolation and Model Reduction

TL;DR

This paper develops a theoretical framework for training neural networks using reproducing kernel space interpolation, introduces Prolongation Neural Networks (PNN), and demonstrates their superior performance in noisy environments.

Contribution

It generalizes neural network training to reproducing kernel Krein spaces and introduces PNNs based on a multidimensional AAK theorem, enhancing expressivity and robustness.

Findings

PNNs outperform existing methods in noisy settings.

Theoretical generalization of neural networks to Krein spaces.

Development of techniques to improve activation function expressivity.

Abstract

We introduce and study the theory of training neural networks using interpolation techniques from reproducing kernel Hilbert space theory. We generalize the method to Krein spaces, and show that widely-used neural network architectures are subsets of reproducing kernel Krein spaces (RKKS). We study the concept of "associated Hilbert spaces" of RKKS and develop techniques to improve upon the expressivity of various activation functions. Next, using concepts from the theory of functions of several complex variables, we prove a computationally applicable, multidimensional generalization of the celebrated Adamjan- Arov-Krein (AAK) theorem. The theorem yields a novel class of neural networks, called Prolongation Neural Networks (PNN). We demonstrate that, by applying the multidimensional AAK theorem to gain a PNN, one can gain performance superior to both our interpolatory methods and current state-of-the-art methods in noisy environments. We provide useful illustrations of our methods in practice.