Steps Toward Deep Kernel Methods from Infinite Neural Networks

TL;DR

This paper explores the connection between deep infinite neural networks and kernel methods, providing theoretical insights into their generalization and stability, and proposing stochastic kernels to encode their information.

Contribution

It establishes a link between deep infinite neural networks and Gaussian processes, introducing stochastic kernels that capture their behavior.

Findings

Deep infinite neural networks align with Gaussian processes and kernel methods.

Stability results apply to large networks, explaining their empirical success.

Proposed stochastic kernels encode information of deep infinite networks.

Abstract

Contemporary deep neural networks exhibit impressive results on practical problems. These networks generalize well although their inherent capacity may extend significantly beyond the number of training examples. We analyze this behavior in the context of deep, infinite neural networks. We show that deep infinite layers are naturally aligned with Gaussian processes and kernel methods, and devise stochastic kernels that encode the information of these networks. We show that stability results apply despite the size, offering an explanation for their empirical success.