A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecture

TL;DR

This paper explores hybrid and evolutionary metaheuristics, specifically PSO and GAs, as alternatives to SGD for training neural networks, demonstrating significant improvements in training efficiency and accuracy.

Contribution

It introduces a hybrid PSO-SGD strategy and evaluates evolutionary algorithms, showing their effectiveness over traditional methods in neural network training.

Findings

Hybrid PSO-SGD reduces median training MSE by 90-95%.

Evolutionary algorithms outperform conventional methods in training accuracy.

Subpar performance of RS highlights the effectiveness of hybrid and evolutionary approaches.

Abstract

Training Artificial Neural Networks (ANNs) with Stochastic Gradient Descent (SGD) frequently encounters difficulties, including substantial computing expense and the risk of converging to local optima, attributable to its dependence on partial weight gradients. Therefore, this work investigates Particle Swarm Optimization (PSO) and Genetic Algorithms (GAs) - two population-based Metaheuristic Optimizers (MHOs) - as alternatives to SGD to mitigate these constraints. A hybrid PSO-SGD strategy is developed to improve local search efficiency. The findings indicate that the hybrid PSO-SGD technique decreases the median training MSE by 90 to 95 percent relative to conventional GA and PSO across various network sizes (e.g., from around 0.02 to approximately 0.001 in the Sphere function). RMHC attains substantial enhancements, reducing MSE by roughly 85 to 90 percent compared to GA. Simultaneously, RS consistently exhibits errors exceeding 0.3, signifying subpar performance. These findings underscore that hybrid and evolutionary procedures significantly improve training efficiency and accuracy compared to conventional optimization methods and imply that the Building Block Hypothesis (BBH) may still be valid, indicating that advantageous weight structures are retained during evolutionary search.