Fast Approximation and Estimation Bounds of Kernel Quadrature for Infinitely Wide Models

TL;DR

This paper introduces a new kernel quadrature method for infinitely wide models, achieving faster approximation and estimation rates, and providing a new complexity measure linked to generalization performance.

Contribution

The paper develops the general kernel quadrature (GKQ) for parameter distributions, surpassing traditional rates and connecting complexity to generalization in deep learning models.

Findings

Achieves exponential approximation rate $O(e^{-p})$ with parameter number $p$.

Attains estimation rate $ ilde{O}(1/n)$ with sample size $n$.

Provides a new norm-based complexity measure for infinitely wide models.

Abstract

An infinitely wide model is a weighted integration $\int \varphi(x,v) d \mu(v)$ of feature maps. This model excels at handling an infinite number of features, and thus it has been adopted to the theoretical study of deep learning. Kernel quadrature is a kernel-based numerical integration scheme developed for fast approximation of expectations $\int f(x) d p(x)$. In this study, regarding the weight $\mu$ as a signed (or complex/vector-valued) distribution of parameters, we develop the general kernel quadrature (GKQ) for parameter distributions. The proposed method can achieve a fast approximation rate $O(e^{-p})$ with parameter number $p$, which is faster than the traditional Barron's rate, and a fast estimation rate $\widetilde{O}(1/n)$ with sample size $n$. As a result, we have obtained a new norm-based complexity measure for infinitely wide models. Since the GKQ implicitly conducts the empirical risk minimization, we can understand that the complexity measure also reflects the generalization performance in the gradient learning setup.