Realization Theory Of Recurrent Neural ODEs Using Polynomial System Embeddings

TL;DR

This paper demonstrates that recurrent neural ODEs like ODE-RNN and ODE-LSTM can be embedded into polynomial systems, enabling the application of realization theory for analysis and potential model reduction.

Contribution

It introduces a polynomial system embedding for neural ODEs and develops realization theory conditions specific to ODE-LSTM architectures.

Findings

Embedded neural ODEs into polynomial systems preserving input-output behavior

Provided necessary conditions for realizability of input-output maps by ODE-LSTM

Established minimality conditions for polynomial system representations

Abstract

In this paper we show that neural ODE analogs of recurrent (ODE-RNN) and Long Short-Term Memory (ODE-LSTM) networks can be algorithmically embeddeded into the class of polynomial systems. This embedding preserves input-output behavior and can suitably be extended to other neural DE architectures. We then use realization theory of polynomial systems to provide necessary conditions for an input-output map to be realizable by an ODE-LSTM and sufficient conditions for minimality of such systems. These results represent the first steps towards realization theory of recurrent neural ODE architectures, which is is expected be useful for model reduction and learning algorithm analysis of recurrent neural ODEs.