Deep Networks are Reproducing Kernel Chains

TL;DR

This paper introduces chain RKBS, a new framework linking deep neural networks with reproducing kernel spaces, showing that deep networks can be characterized as neural cRKBS functions, enabling sparse solutions for empirical risk minimization.

Contribution

It extends RKBS to chain RKBS, establishing a theoretical equivalence between deep neural networks and neural cRKBS functions, and offers a sparse, data-efficient approach to training.

Findings

Deep networks are equivalent to neural cRKBS functions.

Neural cRKBS functions on finite data correspond to deep neural networks.

Sparse solutions require no more than N neurons per layer, where N is data points.

Abstract

Identifying an appropriate function space for deep neural networks remains a key open question. While shallow neural networks are naturally associated with Reproducing Kernel Banach Spaces (RKBS), deep networks present unique challenges. In this work, we extend RKBS to chain RKBS (cRKBS), a new framework that composes kernels rather than functions, preserving the desirable properties of RKBS. We prove that any deep neural network function is a neural cRKBS function, and conversely, any neural cRKBS function defined on a finite dataset corresponds to a deep neural network. This approach provides a sparse solution to the empirical risk minimization problem, requiring no more than $N$ neurons per layer, where $N$ is the number of data points.