PAC-Bayesian bounds for learning LTI-ss systems with input from empirical loss

TL;DR

This paper derives PAC-Bayesian error bounds for linear time-invariant systems with inputs, linking future prediction errors to training data errors, and potentially extending to recurrent neural networks.

Contribution

It introduces a PAC-Bayesian error bound specifically for LTI systems with inputs, enabling finite-sample guarantees for system identification algorithms.

Findings

Provides finite-sample error bounds for LTI system learning algorithms.

Connects future prediction errors with empirical training errors.

Lays groundwork for PAC-Bayesian bounds in RNNs.

Abstract

In this paper we derive a Probably Approxilmately Correct(PAC)-Bayesian error bound for linear time-invariant (LTI) stochastic dynamical systems with inputs. Such bounds are widespread in machine learning, and they are useful for characterizing the predictive power of models learned from finitely many data points. In particular, with the bound derived in this paper relates future average prediction errors with the prediction error generated by the model on the data used for learning. In turn, this allows us to provide finite-sample error bounds for a wide class of learning/system identification algorithms. Furthermore, as LTI systems are a sub-class of recurrent neural networks (RNNs), these error bounds could be a first step towards PAC-Bayesian bounds for RNNs.