Approximate Inference for Spectral Mixture Kernel

TL;DR

This paper introduces an approximate Bayesian inference method for the spectral mixture kernel, enhancing training efficiency and parameter optimization by employing variational distribution, sampling strategies, and natural gradients.

Contribution

It proposes a novel variational inference approach with strategies to improve convergence and reduce overfitting in spectral mixture kernel learning.

Findings

Accelerates convergence of kernel parameter training.

Achieves better kernel parameter optimization.

Reduces overfitting in spectral mixture kernel learning.

Abstract

A spectral mixture (SM) kernel is a flexible kernel used to model any stationary covariance function. Although it is useful in modeling data, the learning of the SM kernel is generally difficult because optimizing a large number of parameters for the SM kernel typically induces an over-fitting, particularly when a gradient-based optimization is used. Also, a longer training time is required. To improve the training, we propose an approximate Bayesian inference for the SM kernel. Specifically, we employ the variational distribution of the spectral points to approximate SM kernel with a random Fourier feature. We optimize the variational parameters by applying a sampling-based variational inference to the derived evidence lower bound (ELBO) estimator constructed from the approximate kernel. To improve the inference, we further propose two additional strategies: (1) a sampling strategy of spectral points to estimate the ELBO estimator reliably and thus its associated gradient, and (2) an approximate natural gradient to accelerate the convergence of the parameters. The proposed inference combined with two strategies accelerates the convergence of the parameters and leads to better optimal parameters.