A Continuous-Time Ensemble Kalman-Bucy Smoother for Causal Inference and Model Discovery

TL;DR

This paper introduces an ensemble Kalman-Bucy smoother (EnKBS) for continuous-time data assimilation that improves state estimation and causal inference in nonlinear dynamical systems without requiring explicit model derivatives.

Contribution

The paper presents a derivative-free, ensemble-based continuous-time smoother that can perform causal inference and model discovery efficiently with small ensembles.

Findings

EnKBS converges to the exact smoother solution at large ensemble sizes.

It effectively performs causal inference with small ensembles under partial observations.

EnKBS supports high-dimensional system discovery over time.

Abstract

Data assimilation (DA) integrates observational information with model predictions to improve state estimation in complex systems. While filtering provides the basis for online forecasts by using only past and present observations, it can exhibit delays and biases when the underlying dynamics evolve rapidly or undergo regime transitions. Smoothing, which additionally incorporates future observations, provides a natural pipeline for hindcasting and reanalysis that yields an uncertainty reduction beyond the filter. This paper introduces an ensemble Kalman-Bucy smoother (EnKBS) for continuous-time DA of nonlinear dynamical systems, where the smoother's conditional distributions are reconstructed using ensemble moments. The result is a derivative-free framework that does not require explicit computation of tangent-linear or adjoint models, which converges to the exact smoother solution at the infinite-ensemble limit for a wide class of complex systems. Incorporating standard regularization techniques for high-dimensional systems, such as covariance localization and inflation, the skill of the EnKBS is demonstrated in various important scientific problems. By integrating future observations, which reveal the underlying causal mechanisms for retrospective state updates, the EnKBS is used for Bayesian-based inference of causal relationships and their temporal influence range in a dyadic trigger-feedback model and the development of a causality-driven iterative learning algorithm that identifies the structure and recovers the hidden parameters of a nonlinear reduced-order model mimicking midlatitude atmospheric circulation. Notably, both tasks remain effective with an ensemble size of $O(10)$ under partial observations, suggesting that EnKBS can support the instantaneous discovery of high-dimensional complex systems over time.