Gradient flow in the kernel learning problem

TL;DR

This paper investigates a Riemannian gradient flow approach for kernel learning, revealing its noise-reduction properties and providing insights into stationary points, with comparisons to neural networks.

Contribution

It introduces a canonical Riemannian gradient flow for kernel learning and analyzes its properties, including noise reduction and stationary point characterization.

Findings

Flow has a family of Lyapunov functionals

Flow automatically reduces noise in the presence of Gaussian variables

Comparison with 2-layer neural networks provided

Abstract

This is a sequel to our paper `On the kernel learning problem'. We identify a canonical choice of Riemannian gradient flow, to find the stationary points in the kernel learning problem. In the presence of Gaussian noise variables, this flow enjoys the remarkable property of having a continuous family of Lyapunov functionals, and the interpretation is the automatic reduction of noise. PS. We include an extensive discussion in the postcript explaining the comparison with the 2-layer neural networks. Readers looking for additional motivations are encouraged to read the postscript immediately following the introduction.