Generic Variance Bounds on Estimation and Prediction Errors in Time Series Analysis: An Entropy Perspective

TL;DR

This paper derives general variance bounds for estimation and prediction errors in time series using an information-theoretic approach, linking error bounds to conditional entropy and characterizing conditions for asymptotic optimality.

Contribution

It introduces a unified entropy-based framework for bounding errors in time series analysis and characterizes the conditions for achieving these bounds asymptotically.

Findings

Error bounds are determined by the conditional entropy of the data point.

Asymptotic optimality requires the innovation to be asymptotically white Gaussian.

Bounds reduce to classical formulas in Gaussian linear prediction.

Abstract

In this paper, we obtain generic bounds on the variances of estimation and prediction errors in time series analysis via an information-theoretic approach. It is seen in general that the error bounds are determined by the conditional entropy of the data point to be estimated or predicted given the side information or past observations. Additionally, we discover that in order to achieve the prediction error bounds asymptotically, the necessary and sufficient condition is that the "innovation" is asymptotically white Gaussian. When restricted to Gaussian processes and 1-step prediction, our bounds are shown to reduce to the Kolmogorov-Szeg\"o formula and Wiener-Masani formula known from linear prediction theory.