Kernel-based linear system identification using augmented Krylov subspaces

TL;DR

This paper introduces a new Krylov subspace method for system identification that efficiently estimates the FIR of linear systems using kernel-based models and augmented subspaces for faster convergence.

Contribution

It develops an augmented Krylov subspace approach that jointly approximates data fit and model complexity, enabling accelerated and cost-effective hyperparameter estimation.

Findings

Accelerated convergence in approximating the data fit quadratic form.

Efficient evaluation of the MLE objective for multiple regularization parameters.

Error bounds demonstrating the effectiveness of the augmentation.

Abstract

We propose a novel Krylov subspace method for estimating the finite impulse response (FIR) of a one-dimensional linear time-invariant systems. The method approximates the system's FIR using a kernel-based formulation combined with hyperparameter selection based on maximum likelihood estimation (MLE), which requires repeated evaluation of two terms: The data fit $\boldsymbol{y}^{\top} (\lambda \boldsymbol{I} + \boldsymbol{A})^{-1} \boldsymbol{y}$ and the model complexity $\log(\det (\lambda \boldsymbol{I} + \boldsymbol{A}))$, where $\boldsymbol{A}$ is a certain positive semidefinite matrix that admits fast matrix--vector products and $\lambda > 0$ is a regularization parameter. Instead of approximating these two quantities separately, we jointly approximate them using a single augmented Krylov subspace for $\boldsymbol{A}$. One major benefit of augmentation is that we obtain accelerated convergence when approximating the data fit quadratic form, through implicit preconditioning. Thanks to the shift invariance of Krylov subspaces, the extracted approximations can be used to evaluate the MLE objective for many values of $\lambda$ at little additional cost. We derive error bounds for the approximations, reflecting the benefits of augmentation demonstrated through multiple numerical experiments.