A Novel Neural Network Training Framework with Data Assimilation

TL;DR

This paper introduces a gradient-free neural network training framework using data assimilation techniques, enabling online and offline learning without relying on gradient calculations, and demonstrates its effectiveness through synthetic function regression tasks.

Contribution

The study proposes a novel gradient-free training framework for neural networks based on data assimilation algorithms like EnKF and ESMDA, offering an alternative to traditional gradient descent methods.

Findings

The proposed framework outperforms gradient descent in synthetic regression tasks.

It enables online and offline training without gradient dependence.

The method shows promise for training various neural network architectures.

Abstract

In recent years, the prosperity of deep learning has revolutionized the Artificial Neural Networks. However, the dependence of gradients and the offline training mechanism in the learning algorithms prevents the ANN for further improvement. In this study, a gradient-free training framework based on data assimilation is proposed to avoid the calculation of gradients. In data assimilation algorithms, the error covariance between the forecasts and observations is used to optimize the parameters. Feedforward Neural Networks (FNNs) are trained by gradient decent, data assimilation algorithms (Ensemble Kalman Filter (EnKF) and Ensemble Smoother with Multiple Data Assimilation (ESMDA)), respectively. ESMDA trains FNN with pre-defined iterations by updating the parameters using all the available observations which can be regard as offline learning. EnKF optimize FNN when new observation available by updating parameters which can be regard as online learning. Two synthetic cases with the regression of a Sine Function and a Mexican Hat function are assumed to validate the effectiveness of the proposed framework. The Root Mean Square Error (RMSE) and coefficient of determination (R2) are used as criteria to assess the performance of different methods. The results show that the proposed training framework performed better than the gradient decent method. The proposed framework provides alternatives for online/offline training the existing ANNs (e.g., Convolutional Neural Networks, Recurrent Neural Networks) without the dependence of gradients.