Solving Approximation Tasks with Greedy Deep Kernel Methods

TL;DR

This paper introduces deep multilayer kernel methods with greedy approximation techniques, enhancing expressiveness and automatic parameter adaptation, and demonstrates their advantages over neural networks in various approximation tasks.

Contribution

The paper proposes deep multilayer kernels with greedy approximation, combining kernel layers and activation functions, and shows their superior approximation performance compared to neural networks.

Findings

Deep kernels automatically adapt shape parameters.

Deep kernel models outperform neural networks in approximation accuracy.

Numerical experiments confirm the effectiveness of deep kernel greedy models.

Abstract

Kernel methods are versatile tools for function approximation and surrogate modeling. In particular, greedy techniques offer computational efficiency and reliability through inherent sparsity and provable convergence. Inspired by the success of deep neural networks and structured deep kernel networks, we consider deep, multilayer kernels for greedy approximation. This multilayer structure, consisting of linear kernel layers and optimizable kernel activation function layers in an alternating fashion, increases the expressiveness of the kernels and thus of the resulting approximants. Compared to standard kernels, deep kernels are able to adapt kernel intrinsic shape parameters automatically, incorporate transformations of the input space and induce a data-dependent reproducing kernel Hilbert space. For this, deep kernels need to be pretrained using a specifically tailored optimization objective. In this work, we not only introduce deep kernel greedy models, but also present numerical investigations and comparisons with neural networks, which clearly show the advantages in terms of approximation accuracies. As applications we consider the approximation of model problems, the prediction of breakthrough curves for reactive flow through porous media and the approximation of solutions for parameterized ordinary differential equation systems.