Self-Organizing Score-based Data Assimilation

TL;DR

This paper introduces a novel framework that extends score-based data assimilation to jointly infer latent states and unknown parameters in complex, high-dimensional dynamical systems, combining classical self-organization techniques with modern SDA.

Contribution

It integrates classical self-organization methods into score-based data assimilation to enable joint inference of states and parameters efficiently.

Findings

Validated on neuroscience and atmospheric models.

Demonstrated scalability with high-dimensional Kolmogorov flow.

Achieved accurate joint inference in complex systems.

Abstract

A state-space model is a statistical framework for inferring latent states from observed time-series data. However, inference with nonlinear and high-dimensional state-space models remains challenging. To this end, an approach based on diffusion models-a powerful class of deep generative models-has been developed, known as Score-based Data Assimilation (SDA). However, SDA cannot be directly applied when the latent-state transition depends on unknown parameters that must be inferred jointly with the latent states. To overcome this limitation, we propose a framework that enables SDA to handle latent states with unknown parameters. A key feature of the proposed method is the incorporation of the self-organization technique, which has been used in classical state-space modeling for the joint estimation of latent states and parameters. By integrating this classical technique into modern SDA, our method enables joint inference of latent states and unknown parameters while maintaining the high training efficiency of SDA. The effectiveness of the proposed approach is validated through numerical experiments on dynamical systems arising in neuroscience and atmospheric science. In addition, its scalability is demonstrated using a high-dimensional Kolmogorov flow, with the data dimension on the order of several hundred thousand.