Optimizing Deep Neural Networks through Neuroevolution with Stochastic Gradient Descent

TL;DR

This paper introduces a hybrid neuroevolution and stochastic gradient descent method to optimize deep neural networks, improving performance and diversity over traditional SGD and state-of-the-art models.

Contribution

It presents a novel combined approach of neuroevolution and SGD with a hierarchical suppression algorithm for better DNN optimization.

Findings

Optimized DNNs outperform those trained with only SGD.

The approach surpasses state-of-the-art deep networks.

Enhanced population diversity improves optimization results.

Abstract

Deep neural networks (DNNs) have achieved remarkable success in computer vision; however, training DNNs for satisfactory performance remains challenging and suffers from sensitivity to empirical selections of an optimization algorithm for training. Stochastic gradient descent (SGD) is dominant in training a DNN by adjusting neural network weights to minimize the DNNs loss function. As an alternative approach, neuroevolution is more in line with an evolutionary process and provides some key capabilities that are often unavailable in SGD, such as the heuristic black-box search strategy based on individual collaboration in neuroevolution. This paper proposes a novel approach that combines the merits of both neuroevolution and SGD, enabling evolutionary search, parallel exploration, and an effective probe for optimal DNNs. A hierarchical cluster-based suppression algorithm is also developed to overcome similar weight updates among individuals for improving population diversity. We implement the proposed approach in four representative DNNs based on four publicly-available datasets. Experiment results demonstrate that the four DNNs optimized by the proposed approach all outperform corresponding ones optimized by only SGD on all datasets. The performance of DNNs optimized by the proposed approach also outperforms state-of-the-art deep networks. This work also presents a meaningful attempt for pursuing artificial general intelligence.