Maximum Entropy Kernels for System Identification

TL;DR

This paper demonstrates that the Diagonal/Correlated (DC) kernels used in nonparametric system identification possess maximum entropy properties, enabling closed-form factorization and improved computational efficiency.

Contribution

It extends maximum entropy properties to the entire family of DC kernels and provides closed-form expressions for their factorization, inverse, and determinant.

Findings

DC kernels have maximum entropy properties.

Closed-form factorization, inverse, and determinant of DC kernels.

Results improve numerical stability and reduce computational complexity.

Abstract

A new nonparametric approach for system identification has been recently proposed where the impulse response is modeled as the realization of a zero-mean Gaussian process whose covariance (kernel) has to be estimated from data. In this scheme, quality of the estimates crucially depends on the parametrization of the covariance of the Gaussian process. A family of kernels that have been shown to be particularly effective in the system identification framework is the family of Diagonal/Correlated (DC) kernels. Maximum entropy properties of a related family of kernels, the Tuned/Correlated (TC) kernels, have been recently pointed out in the literature. In this paper we show that maximum entropy properties indeed extend to the whole family of DC kernels. The maximum entropy interpretation can be exploited in conjunction with results on matrix completion problems in the graphical models literature to shed light on the structure of the DC kernel. In particular, we prove that the DC kernel admits a closed-form factorization, inverse and determinant. These results can be exploited both to improve the numerical stability and to reduce the computational complexity associated with the computation of the DC estimator.