# Optimization Based Methods for Partially Observed Chaotic Systems

## TL;DR

This paper develops a theoretical framework for filtering and smoothing in partially observed chaotic systems, demonstrating asymptotic optimality of MAP estimators and providing efficient algorithms, with applications to high-dimensional Lorenz 96' models.

## Contribution

It introduces a Gaussian approximation for filters and smoothers in chaotic systems, proving MAP estimator optimality and offering a fast Newton-based algorithm for high-dimensional models.

## Key findings

- Gaussian approximation is valid for small h and noise variance
- MAP estimators are asymptotically optimal as noise diminishes
- Efficient Newton-based algorithm converges rapidly in high dimensions

## Abstract

In this paper we consider filtering and smoothing of partially observed chaotic dynamical systems that are discretely observed, with an additive Gaussian noise in the observation. These models are found in a wide variety of real applications and include the Lorenz 96' model. In the context of a fixed observation interval $T$, observation time step $h$ and Gaussian observation variance $\sigma_Z^2$, we show under assumptions that the filter and smoother are well approximated by a Gaussian with high probability when $h$ and $\sigma^2_Z h$ are sufficiently small. Based on this result we show that the Maximum-a-posteriori (MAP) estimators are asymptotically optimal in mean square error as $\sigma^2_Z h$ tends to $0$. Given these results, we provide a batch algorithm for the smoother and filter, based on Newton's method, to obtain the MAP. In particular, we show that if the initial point is close enough to the MAP, then Newton's method converges to it at a fast rate. We also provide a method for computing such an initial point. These results contribute to the theoretical understanding of widely used 4D-Var data assimilation method. Our approach is illustrated numerically on the Lorenz 96' model with state vector up to 1 million dimensions, with code running in the order of minutes. To our knowledge the results in this paper are the first of their type for this class of models.