Relative entropy minimizing noisy non-linear neural network to approximate stochastic processes

TL;DR

This paper introduces a unified method for designing noise-driven recurrent neural networks that approximate stochastic processes by minimizing relative entropy, demonstrated on climate-related stochastic modeling.

Contribution

It unifies Echo State Networks and Linear Inverse Modeling under a common entropy minimization framework, advancing stochastic process modeling.

Findings

Successfully modeled El Nino phenomenon using the proposed method.

Demonstrated the effectiveness of the approach in climate stochastic modeling.

Generalized existing neural network approaches for stochastic process approximation.

Abstract

A method is provided for designing and training noise-driven recurrent neural networks as models of stochastic processes. The method unifies and generalizes two known separate modeling approaches, Echo State Networks (ESN) and Linear Inverse Modeling (LIM), under the common principle of relative entropy minimization. The power of the new method is demonstrated on a stochastic approximation of the El Nino phenomenon studied in climate research.