Auto-differentiable data assimilation: Co-learning of states, dynamics, and filtering algorithms

TL;DR

This paper presents a unified framework for jointly learning states, dynamics, and filtering parameters in data assimilation using auto-differentiation, improving tuning and model accuracy across various scientific domains.

Contribution

It introduces auto-differentiable filtering, enabling simultaneous learning of states, dynamics, and parameters, which enhances data assimilation methods.

Findings

Framework successfully learns and tunes data assimilation algorithms.

Demonstrated versatility across aerospace, atmospheric, and biological systems.

Provides practical guidelines for customization and implementation.

Abstract

Data assimilation algorithms estimate the state of a dynamical system from partial observations, where the successful performance of these algorithms hinges on costly parameter tuning and on employing an accurate model for the dynamics. This paper introduces a framework for jointly learning the state, dynamics, and parameters of filtering algorithms in data assimilation through a process we refer to as auto-differentiable filtering. The framework leverages a theoretically motivated loss function that enables learning from partial, noisy observations via gradient-based optimization using auto-differentiation. We further demonstrate how several well-known data assimilation methods can be learned or tuned within this framework. To underscore the versatility of auto-differentiable filtering, we perform experiments on dynamical systems spanning multiple scientific domains, such as the Clohessy-Wiltshire equations from aerospace engineering, the Lorenz-96 system from atmospheric science, and the generalized Lotka-Volterra equations from systems biology. Finally, we provide guidelines for practitioners to customize our framework according to their observation model, accuracy requirements, and computational budget.