Reconstruction of Stochastic Dynamics from Large Streamed Datasets

TL;DR

This paper introduces an online method for estimating stochastic differential equation parameters from large datasets, enabling analysis of complex systems and real-time data streams efficiently.

Contribution

The paper presents a novel incremental approach for estimating drift and diffusion functions from large-scale, streamed datasets, overcoming computational limitations of traditional methods.

Findings

Validated on synthetic datasets showing accurate estimation

Successfully applied to empirical turbulence data

Facilitates real-time analysis of big data in complex systems

Abstract

The complex dynamics of physical systems can often be modeled with stochastic differential equations. However, computational constraints inhibit the estimation of dynamics from large time-series datasets. I present a method for estimating drift and diffusion functions from inordinately large datasets through the use of incremental, online, updating statistics. I demonstrate the validity and utility of this method by analyzing three large, varied synthetic datasets, as well as an empirical turbulence dataset. This method will hopefully facilitate the analysis of complex systems from exceedingly large, "big data" scientific datasets, as well as real-time streamed data.