Surrogate-Assisted Evolution for Efficient Multi-branch Connection Design in Deep Neural Networks

TL;DR

This paper presents a surrogate-assisted evolutionary approach using Linear Genetic Programming to efficiently design multi-branch deep neural networks, significantly reducing computational costs while discovering high-performing architectures.

Contribution

It introduces NeuroLGP-MB, a novel encoding for multi-branch connections, and advances surrogate-assisted EAs with semantic-based modeling for complex DNN architecture search.

Findings

Surrogate-assisted EAs scale to thousands of samples for DNN design.

The proposed surrogate model outperforms baseline and simpler models.

Efficiently discovers high-performing multi-branch DNN architectures.

Abstract

State-of-the-art Deep Neural Networks (DNNs) often incorporate multi-branch connections, enabling multi-scale feature extraction and enhancing the capture of diverse features. This design improves network capacity and generalisation to unseen data. However, training such DNNs can be computationally expensive. The challenge is further exacerbated by the complexity of identifying optimal network architectures. To address this, we leverage Evolutionary Algorithms (EAs) to automatically discover high-performing architectures, a process commonly known as neuroevolution. We introduce a novel approach based on Linear Genetic Programming (LGP) to encode multi-branch (MB) connections within DNNs, referred to as NeuroLGP-MB. To efficiently design the DNNs, we use surrogate-assisted EAs. While their application in simple artificial neural networks has been influential, we scale their use from dozens or hundreds of sample points to thousands, aligning with the demands of complex DNNs by incorporating a semantic-based approach in our surrogate-assisted EA. Furthermore, we introduce a more advanced surrogate model that outperforms baseline, computationally expensive, and simpler surrogate models.