Random walk kernels and learning curves for Gaussian process regression on random graphs

TL;DR

This paper investigates the properties of random walk kernels on large, locally treelike graphs, highlighting issues with global normalization and proposing local normalization to improve Gaussian process regression models, supported by theoretical and numerical analysis.

Contribution

It introduces a local normalization method for random walk kernels on graphs and analyzes its impact on Gaussian process learning curves, providing more accurate predictions via belief propagation.

Findings

Global normalization causes uneven prior variances across vertices.

Local normalization equalizes prior variances, leading to better probabilistic models.

Belief propagation yields highly accurate learning curve predictions.

Abstract

We consider learning on graphs, guided by kernels that encode similarity between vertices. Our focus is on random walk kernels, the analogues of squared exponential kernels in Euclidean spaces. We show that on large, locally treelike, graphs these have some counter-intuitive properties, specifically in the limit of large kernel lengthscales. We consider using these kernels as covariance matrices of e.g.\ Gaussian processes (GPs). In this situation one typically scales the prior globally to normalise the average of the prior variance across vertices. We demonstrate that, in contrast to the Euclidean case, this generically leads to significant variation in the prior variance across vertices, which is undesirable from the probabilistic modelling point of view. We suggest the random walk kernel should be normalised locally, so that each vertex has the same prior variance, and analyse the consequences of this by studying learning curves for Gaussian process regression. Numerical calculations as well as novel theoretical predictions for the learning curves using belief propagation make it clear that one obtains distinctly different probabilistic models depending on the choice of normalisation. Our method for predicting the learning curves using belief propagation is significantly more accurate than previous approximations and should become exact in the limit of large random graphs.