An Empirical Bayes approach to ARX Estimation

TL;DR

This paper compares marginal and full Empirical Bayes methods for estimating ARX models in time series, introducing a new hyperparameter estimation technique using a backward Kalman filter, with marginal Bayes performing slightly better on finite data.

Contribution

It introduces a sequential Bayes hyperparameter estimation method for ARX models and compares it with full Empirical Bayesian analysis, highlighting performance differences.

Findings

Marginal Bayes estimator slightly outperforms full Empirical Bayesian in finite data scenarios.

Proposes a backward Kalman filter-based hyperparameter estimation technique.

Empirical comparison on ARX models demonstrates practical advantages of marginal approach.

Abstract

Empirical Bayes inference is based on estimation of the parameters of an a priori distribution from the observed data. The estimation technique of the parameters of the prior, called hyperparameters, is based on the marginal distribution obtained by integrating the joint density of the model with respect to the prior. This is a key step which needs to be properly adapted to the problem at hand. In this paper we study Empirical Bayes inference of linear autoregressive models with inputs (ARX models) for time series and compare the performance of the marginal parametric estimator with that a full Empirical Bayesian analysis based on the estimated prior. Such a comparison, can only make sense for a (realistic) finite data length. In this setting, we propose a new estimation technique of the hyperparameters by a sequential Bayes procedure which is essentially a backward Kalman filter. It turns out that for finite data length the marginal Bayes tends to behave slightly better than the full Empirical Bayesian parameter estimator and so also in the case of slowly varying random parameters.