Improved Initialization of State-Space Artificial Neural Networks

TL;DR

This paper presents an improved method for initializing neural network parameters in nonlinear state-space models, enhancing convergence and performance in system identification tasks.

Contribution

It introduces a novel initialization technique that combines linear approximation-based weights with random or zero values, emphasizing the role of explicit linear terms.

Findings

Improved initialization leads to better convergence in nonlinear system identification.

The proposed method outperforms previous initialization approaches on benchmark examples.

Including an explicit linear term enhances model accuracy and training stability.

Abstract

The identification of black-box nonlinear state-space models requires a flexible representation of the state and output equation. Artificial neural networks have proven to provide such a representation. However, as in many identification problems, a nonlinear optimization problem needs to be solved to obtain the model parameters (layer weights and biases). A well-thought initialization of these model parameters can often avoid that the nonlinear optimization algorithm converges to a poorly performing local minimum of the considered cost function. This paper introduces an improved initialization approach for nonlinear state-space models represented as a recurrent artificial neural network and emphasizes the importance of including an explicit linear term in the model structure. Some of the neural network weights are initialized starting from a linear approximation of the nonlinear system, while others are initialized using random values or zeros. The effectiveness of the proposed initialization approach over previously proposed methods is illustrated on two benchmark examples.