Evolving Parsimonious Networks by Mixing Activation Functions

TL;DR

This paper extends the NEAT neuroevolution algorithm to evolve activation functions of neurons, demonstrating that heterogeneous networks with mixed activation functions can be smaller and perform well on regression and classification tasks.

Contribution

It introduces a method to evolve activation functions within NEAT, enabling the automatic selection of effective functions and producing more compact networks.

Findings

Heterogeneous networks outperform homogeneous ones in size and performance.

NEAT can effectively select suitable activation functions during evolution.

Heterogeneous networks are significantly smaller than homogeneous networks.

Abstract

Neuroevolution methods evolve the weights of a neural network, and in some cases the topology, but little work has been done to analyze the effect of evolving the activation functions of individual nodes on network size, which is important when training networks with a small number of samples. In this work we extend the neuroevolution algorithm NEAT to evolve the activation function of neurons in addition to the topology and weights of the network. The size and performance of networks produced using NEAT with uniform activation in all nodes, or homogenous networks, is compared to networks which contain a mixture of activation functions, or heterogenous networks. For a number of regression and classification benchmarks it is shown that, (1) qualitatively different activation functions lead to different results in homogeneous networks, (2) the heterogeneous version of NEAT is able to select well performing activation functions, (3) producing heterogeneous networks that are significantly smaller than homogeneous networks.