An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems

TL;DR

This paper introduces an ensemble score filter (EnSF) that leverages a training-free score estimation method based on diffusion models to improve accuracy in high-dimensional nonlinear filtering, outperforming existing methods like particle filters and ensemble Kalman filters.

Contribution

The paper presents a novel training-free score estimation technique within an ensemble score filter, enabling efficient high-dimensional nonlinear filtering without neural network training.

Findings

EnSF accurately tracks high-dimensional Lorenz systems up to 1,000,000 dimensions.

EnSF outperforms the Local Ensemble Transform Kalman Filter in nonlinear, high-dimensional scenarios.

The method significantly reduces training time while maintaining high filtering accuracy.

Abstract

We propose an ensemble score filter (EnSF) for solving high-dimensional nonlinear filtering problems with superior accuracy. A major drawback of existing filtering methods, e.g., particle filters or ensemble Kalman filters, is the low accuracy in handling high-dimensional and highly nonlinear problems. EnSF attacks this challenge by exploiting the score-based diffusion model, defined in a pseudo-temporal domain, to characterizing the evolution of the filtering density. EnSF stores the information of the recursively updated filtering density function in the score function, instead of storing the information in a set of finite Monte Carlo samples (used in particle filters and ensemble Kalman filters). Unlike existing diffusion models that train neural networks to approximate the score function, we develop a training-free score estimation that uses a mini-batch-based Monte Carlo estimator to directly approximate the score function at any pseudo-spatial-temporal location, which provides sufficient accuracy in solving high-dimensional nonlinear problems as well as saves a tremendous amount of time spent on training neural networks. High-dimensional Lorenz-96 systems are used to demonstrate the performance of our method. EnSF provides surprising performance, compared with the state-of-the-art Local Ensemble Transform Kalman Filter method, in reliably and efficiently tracking extremely high-dimensional Lorenz systems (up to 1,000,000 dimensions) with highly nonlinear observation processes.