An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution

TL;DR

This study investigates how different weight initialization methods and Lamarckian weight inheritance influence the efficiency and effectiveness of neuroevolution in evolving recurrent neural networks, demonstrating that Lamarckian strategies significantly improve speed and performance.

Contribution

It introduces and evaluates Lamarckian weight inheritance methods within neuroevolution, showing they outperform traditional initialization techniques in evolving RNNs.

Findings

Lamarckian strategies outperform traditional initialization methods.

Lamarckian methods reduce the number of backpropagation epochs needed.

Statistically significant improvements in neuroevolution speed and performance.

Abstract

Weight initialization is critical in being able to successfully train artificial neural networks (ANNs), and even more so for recurrent neural networks (RNNs) which can easily suffer from vanishing and exploding gradients. In neuroevolution, where evolutionary algorithms are applied to neural architecture search, weights typically need to be initialized at three different times: when initial genomes (ANN architectures) are created at the beginning of the search, when offspring genomes are generated by crossover, and when new nodes or edges are created during mutation. This work explores the difference between using Xavier, Kaiming, and uniform random weight initialization methods, as well as novel Lamarckian weight inheritance methods for initializing new weights during crossover and mutation operations. These are examined using the Evolutionary eXploration of Augmenting Memory Models (EXAMM) neuroevolution algorithm, which is capable of evolving RNNs with a variety of modern memory cells (e.g., LSTM, GRU, MGU, UGRNN and Delta-RNN cells) as well recurrent connections with varying time skips through a high performance island based distributed evolutionary algorithm. Results show that with statistical significance, utilizing the Lamarckian strategies outperforms Kaiming, Xavier and uniform random weight initialization, and can speed neuroevolution by requiring less backpropagation epochs to be evaluated for each generated RNN.