The interplay between system identification and machine learning

TL;DR

This paper bridges system identification and machine learning by introducing RKHSs of dynamic systems, enabling better understanding, stability analysis, and convergence guarantees for kernel-based estimators in dynamic contexts.

Contribution

It introduces RKHSs tailored for dynamic systems, linking system identification with machine learning, and provides stability conditions and convergence results for these new kernels.

Findings

RKHSs of dynamic systems enable interpretation of system identification as learning.

Conditions for RKHS stability facilitate the design of new kernels.

Regularized estimators converge to the optimal predictor under typical system conditions.

Abstract

Learning from examples is one of the key problems in science and engineering. It deals with function reconstruction from a finite set of direct and noisy samples. Regularization in reproducing kernel Hilbert spaces (RKHSs) is widely used to solve this task and includes powerful estimators such as regularization networks. Recent achievements include the proof of the statistical consistency of these kernel- based approaches. Parallel to this, many different system identification techniques have been developed but the interaction with machine learning does not appear so strong yet. One reason is that the RKHSs usually employed in machine learning do not embed the information available on dynamic systems, e.g. BIBO stability. In addition, in system identification the independent data assumptions routinely adopted in machine learning are never satisfied in practice. This paper provides new results which strengthen the connection between system identification and machine learning. Our starting point is the introduction of RKHSs of dynamic systems. They contain functionals over spaces defined by system inputs and allow to interpret system identification as learning from examples. In both linear and nonlinear settings, it is shown that this perspective permits to derive in a relatively simple way conditions on RKHS stability (i.e. the property of containing only BIBO stable systems or predictors), also facilitating the design of new kernels for system identification. Furthermore, we prove the convergence of the regularized estimator to the optimal predictor under conditions typical of dynamic systems.