Duality for Neural Networks through Reproducing Kernel Banach Spaces

TL;DR

This paper introduces a novel duality framework for neural networks using Reproducing Kernel Banach Spaces, enabling better understanding of their structure and facilitating primal-dual optimization methods.

Contribution

It extends Barron spaces into Reproducing Kernel Banach Spaces, establishing a duality that interchanges data and parameters, and enabling saddle point formulations for neural network training.

Findings

Barron spaces are shown to be a subset of integral RKBS.

The dual of an RKBS is also an RKBS with roles of data and parameters interchanged.

A saddle point problem for neural networks is constructed using RKBS duality.

Abstract

Reproducing Kernel Hilbert spaces (RKHS) have been a very successful tool in various areas of machine learning. Recently, Barron spaces have been used to prove bounds on the generalisation error for neural networks. Unfortunately, Barron spaces cannot be understood in terms of RKHS due to the strong nonlinear coupling of the weights. This can be solved by using the more general Reproducing Kernel Banach spaces (RKBS). We show that these Barron spaces belong to a class of integral RKBS. This class can also be understood as an infinite union of RKHS spaces. Furthermore, we show that the dual space of such RKBSs, is again an RKBS where the roles of the data and parameters are interchanged, forming an adjoint pair of RKBSs including a reproducing kernel. This allows us to construct the saddle point problem for neural networks, which can be used in the whole field of primal-dual optimisation.