Genetically Modified Wolf Optimization with Stochastic Gradient Descent for Optimising Deep Neural Networks

TL;DR

This paper introduces GMW-SGD, a hybrid optimization algorithm combining genetic modifications of Grey Wolf Optimizer with SGD, aimed at improving neural network training by balancing exploration and exploitation.

Contribution

It presents a novel hybrid metaheuristic algorithm that enhances neural network optimization by integrating genetic modifications with GWO and SGD.

Findings

GMW-SGD achieves comparable accuracy to SGD on CIFAR-10.

It outperforms standard metaheuristic algorithms in neural network optimization.

The method effectively balances exploration and exploitation in high-dimensional spaces.

Abstract

When training Convolutional Neural Networks (CNNs) there is a large emphasis on creating efficient optimization algorithms and highly accurate networks. The state-of-the-art method of optimizing the networks is done by using gradient descent algorithms, such as Stochastic Gradient Descent (SGD). However, there are some limitations presented when using gradient descent methods. The major drawback is the lack of exploration, and over-reliance on exploitation. Hence, this research aims to analyze an alternative approach to optimizing neural network (NN) weights, with the use of population-based metaheuristic algorithms. A hybrid between Grey Wolf Optimizer (GWO) and Genetic Algorithms (GA) is explored, in conjunction with SGD; producing a Genetically Modified Wolf optimization algorithm boosted with SGD (GMW-SGD). This algorithm allows for a combination between exploitation and exploration, whilst also tackling the issue of high-dimensionality, affecting the performance of standard metaheuristic algorithms. The proposed algorithm was trained and tested on CIFAR-10 where it performs comparably to the SGD algorithm, reaching high test accuracy, and significantly outperforms standard metaheuristic algorithms.