Deep Learning with Kernels through RKHM and the Perron-Frobenius Operator

TL;DR

This paper introduces deep RKHM, a novel deep learning framework using kernel methods and $C^*$-algebras, providing new theoretical insights and bounds related to overfitting and model complexity.

Contribution

It combines RKHM and Perron-Frobenius operators to develop deep kernel methods with improved generalization bounds and theoretical understanding.

Findings

Derived a new Rademacher generalization bound for deep RKHM.

Provided a theoretical interpretation of benign overfitting using Perron-Frobenius operators.

Showed that $C^*$-algebra reduces dependency on output dimension in bounds.

Abstract

Reproducing kernel Hilbert $C^*$-module (RKHM) is a generalization of reproducing kernel Hilbert space (RKHS) by means of $C^*$-algebra, and the Perron-Frobenius operator is a linear operator related to the composition of functions. Combining these two concepts, we present deep RKHM, a deep learning framework for kernel methods. We derive a new Rademacher generalization bound in this setting and provide a theoretical interpretation of benign overfitting by means of Perron-Frobenius operators. By virtue of $C^*$-algebra, the dependency of the bound on output dimension is milder than existing bounds. We show that $C^*$-algebra is a suitable tool for deep learning with kernels, enabling us to take advantage of the product structure of operators and to provide a clear connection with convolutional neural networks. Our theoretical analysis provides a new lens through which one can design and analyze deep kernel methods.