Learning to imitate stochastic time series in a compositional way by chaos

TL;DR

This paper demonstrates that a mixture of RNN experts can learn to generate complex stochastic sequences by leveraging chaos, with each expert capturing primitive patterns and the gating network switching among them chaotically.

Contribution

The study introduces a method where a mixture of RNN experts uses chaotic dynamics to imitate stochastic switching among primitive sequence patterns, advancing sequence modeling techniques.

Findings

The model learns primitive patterns through individual experts.

Chaotic dynamics enable stochastic switching among primitives.

Adding small noise stabilizes the sequence reconstruction.

Abstract

This study shows that a mixture of RNN experts model can acquire the ability to generate sequences combining multiple primitive patterns by means of self-organizing chaos. By training of the model, each expert learns a primitive sequence pattern, and a gating network learns to imitate stochastic switching of the multiple primitives via a chaotic dynamics, utilizing a sensitive dependence on initial conditions. As a demonstration, we present a numerical simulation in which the model learns Markov chain switching among some Lissajous curves by a chaotic dynamics. Our analysis shows that by using a sufficient amount of training data, balanced with the network memory capacity, it is possible to satisfy the conditions for embedding the target stochastic sequences into a chaotic dynamical system. It is also shown that reconstruction of a stochastic time series by a chaotic model can be stabilized by adding a negligible amount of noise to the dynamics of the model.