Localized covariance estimation: A Bayesian perspective

TL;DR

This paper applies Bayesian methods to improve high-dimensional covariance matrix estimation in numerical weather prediction, providing theoretical insights and practical guidance for more reliable and less tuned estimators.

Contribution

It offers a Bayesian interpretation of existing covariance estimators and emphasizes the importance of penalizing conditional correlations for better estimation.

Findings

Hybrid estimator has a Bayesian interpretation

Schur product estimator can be analyzed within Bayesian framework

Bayesian approach reduces tuning and improves covariance estimation

Abstract

A major problem in numerical weather prediction (NWP) is the estimation of high-dimensional covariance matrices from a small number of samples. Maximum likelihood estimators cannot provide reliable estimates when the overall dimension is much larger than the number of samples. Fortunately, NWP practitioners have found ingenious ways to boost the accuracy of their covariance estimators by leveraging the assumption that the correlations decay with spatial distance. In this work, Bayesian statistics is used to provide a new justification and analysis of the practical NWP covariance estimators. The Bayesian framework involves manipulating distributions over symmetric positive definite matrices, and it leads to two main findings: (i) the commonly used "hybrid estimator" for the covariance matrix has a naturally Bayesian interpretation; (ii) the very commonly used "Schur product estimator" is not Bayesian, but it can be studied and understood within the Bayesian framework. As practical implications, the Bayesian framework shows how to reduce the amount of tuning required for covariance estimation, and it suggests that efficient covariance estimation should be rooted in understanding and penalizing conditional correlations, rather than correlations.