Regularization and Bayesian Learning in Dynamical Systems: Past, Present and Future

TL;DR

This paper reviews the evolution of regularization and Bayesian methods in system identification, emphasizing historical and foundational issues to clarify their development and connection to past work.

Contribution

It provides a historical perspective on regularization and Bayesian approaches in system identification, highlighting fundamental issues and clarifying links with earlier research.

Findings

Historical evolution of regularization in system identification

Fundamental issues like compound estimation and exchangeability

Clarification of links between recent and past work

Abstract

Regularization and Bayesian methods for system identification have been repopularized in the recent years, and proved to be competitive w.r.t. classical parametric approaches. In this paper we shall make an attempt to illustrate how the use of regularization in system identification has evolved over the years, starting from the early contributions both in the Automatic Control as well as Econometrics and Statistics literature. In particular we shall discuss some fundamental issues such as compound estimation problems and exchangeability which play and important role in regularization and Bayesian approaches, as also illustrated in early publications in Statistics. The historical and foundational issues will be given more emphasis (and space), at the expense of the more recent developments which are only briefly discussed. The main reason for such a choice is that, while the recent literature is readily available, and surveys have already been published on the subject, in the author's opinion a clear link with past work had not been completely clarified.