Mitigating Architectural Mismatch During the Evolutionary Synthesis of Deep Neural Networks

TL;DR

This paper investigates the impact of architectural alignment in evolutionary synthesis of deep neural networks, finding that enforcing alignment affects performance decline and network variability, with comparable final results.

Contribution

It introduces a gene tagging system to enforce architectural alignment during evolutionary synthesis, highlighting its effects on network performance and diversity.

Findings

Slower performance and size reduction with gene tagging

Networks are comparable in size and accuracy to non-gene tagged networks

Gene tagging reduces network variability, limiting search space exploration

Abstract

Evolutionary deep intelligence has recently shown great promise for producing small, powerful deep neural network models via the organic synthesis of increasingly efficient architectures over successive generations. Existing evolutionary synthesis processes, however, have allowed the mating of parent networks independent of architectural alignment, resulting in a mismatch of network structures. We present a preliminary study into the effects of architectural alignment during evolutionary synthesis using a gene tagging system. Surprisingly, the network architectures synthesized using the gene tagging approach resulted in slower decreases in performance accuracy and storage size; however, the resultant networks were comparable in size and performance accuracy to the non-gene tagging networks. Furthermore, we speculate that there is a noticeable decrease in network variability for networks synthesized with gene tagging, indicating that enforcing a like-with-like mating policy potentially restricts the exploration of the search space of possible network architectures.