Optimality vs Stability Trade-off in Ensemble Kalman Filters

TL;DR

This paper investigates the balance between optimality and stability in ensemble Kalman filters, revealing that the most optimal control law may lead to instability, which has implications for high-dimensional data assimilation.

Contribution

It analyzes the relationship between optimal control laws derived from transportation problems and the stability of ensemble Kalman filters, highlighting a trade-off.

Findings

Optimal control laws can cause instability in EnKF algorithms.

There is a fundamental trade-off between optimality and stability in EnKF.

Stability issues arise when using transportation-inspired control laws.

Abstract

This paper is concerned with optimality and stability analysis of a family of ensemble Kalman filter (EnKF) algorithms. EnKF is commonly used as an alternative to the Kalman filter for high-dimensional problems, where storing the covariance matrix is computationally expensive. The algorithm consists of an ensemble of interacting particles driven by a feedback control law. The control law is designed such that, in the linear Gaussian setting and asymptotic limit of infinitely many particles, the mean and covariance of the particles follow the exact mean and covariance of the Kalman filter. The problem of finding a control law that is exact does not have a unique solution, reminiscent of the problem of finding a transport map between two distributions. A unique control law can be identified by introducing control cost functions, that are motivated by the optimal transportation problem or Schr\"odinger bridge problem. The objective of this paper is to study the relationship between optimality and long-term stability of a family of exact control laws. Remarkably, the control law that is optimal in the optimal transportation sense leads to an EnKF algorithm that is not stable.