Laplace Approximation For Tensor Train Kernel Machines In System Identification

TL;DR

This paper introduces a Bayesian tensor train kernel machine that uses Laplace approximation and variational inference to improve scalability and efficiency in Gaussian process regression for system identification.

Contribution

It proposes a novel probabilistic tensor train model with Laplace approximation and VI, addressing core selection and significantly speeding up training.

Findings

VI replaces cross-validation for hyperparameter tuning

Training speed is up to 65 times faster

Core selection is largely independent of TT-ranks

Abstract

To address the scalability limitations of Gaussian process (GP) regression, several approximation techniques have been proposed. One such method is based on tensor networks, which utilizes an exponential number of basis functions without incurring exponential computational cost. However, extending this model to a fully probabilistic formulation introduces several design challenges. In particular, for tensor train (TT) models, it is unclear which TT-core should be treated in a Bayesian manner. We introduce a Bayesian tensor train kernel machine that applies Laplace approximation to estimate the posterior distribution over a selected TT-core and employs variational inference (VI) for precision hyperparameters. Experiments show that core selection is largely independent of TT-ranks and feature structure, and that VI replaces cross-validation while offering up to 65x faster training. The method's effectiveness is demonstrated on an inverse dynamics problem.