Stability properties of Minimal Gated Unit neural networks

TL;DR

This paper analyzes the stability of the Minimal Gated Unit (MGU) neural network, deriving conditions for stability, proposing training methods, and demonstrating its effectiveness and efficiency on synthetic and real-world data, including the Silverbox benchmark.

Contribution

It introduces parametric stability conditions for MGU networks and develops stability-promoting training methods, enhancing efficiency and reliability in system identification tasks.

Findings

MGU networks satisfy input-to-state stability conditions.

Proposed training schemes improve stability and convergence.

MGU outperforms other RNNs on the Silverbox benchmark.

Abstract

In this work, we address the need for efficient and formally stable Recurrent Neural Networks (RNNs) in environments with limited computational resources by analyzing the stability of the Minimal Gated Unit (MGU) network, a lightweight alternative to common gated RNNs used in system identification. We derive sufficient parametric conditions for the MGU network's input-to-state stability and incremental input-to-state stability properties. These conditions enable a-posteriori validation of model stability and form the basis for novel stability-promoting training methodologies, including a warm-start of the network's parameters and a projected gradient-based optimization scheme, both of which are presented in this work. Comparative evaluation, including robustness analysis and validation on synthetic and real-world data (i.e., the Silverbox benchmark), demonstrates that the minimal gated unit network successfully combines formal stability guarantees with superior parameter efficiency and faster inference times compared to other state-of-the-art recurrent neural networks, while maintaining comparable and satisfactory accuracy. Notably, the results attained on the Silverbox benchmark illustrate that the stable MGU network effectively captures the system dynamics, whereas other stable RNNs fail to converge to a reliable model.