Analysis of Structured Deep Kernel Networks

TL;DR

This paper introduces Structured Deep Kernel Networks (SDKNs) that blend kernel methods and neural networks, offering theoretical insights and universal approximation capabilities with potential computational advantages.

Contribution

The paper presents SDKNs as a novel class of models connecting kernel methods and neural networks, with proven approximation properties and efficiency benefits.

Findings

SDKNs can be viewed as neural networks with optimizable activation functions.

SDKNs exhibit universal approximation properties in various asymptotic regimes.

Fewer layers are needed for SDKNs compared to ReLU networks at unbounded depth.

Abstract

In this paper, we leverage a recent deep kernel representer theorem to connect kernel based learning and (deep) neural networks in order to understand their interplay. In particular, we show that the use of special types of kernels yields models reminiscent of neural networks that are founded in the same theoretical framework of classical kernel methods, while benefiting from the computational advantages of deep neural networks. Especially the introduced Structured Deep Kernel Networks (SDKNs) can be viewed as neural networks (NNs) with optimizable activation functions obeying a representer theorem. This link allows us to analyze also NNs within the framework of kernel networks. We prove analytic properties of the SDKNs which show their universal approximation properties in three different asymptotic regimes of unbounded number of centers, width and depth. Especially in the case of unbounded depth, more accurate constructions can be achieved using fewer layers compared to corresponding constructions for ReLU neural networks. This is made possible by leveraging properties of kernel approximation.