Spectrum-Diverse Neuroevolution with Unified Neural Models

TL;DR

This paper introduces a unified neural network representation and a novel diversity preservation method called Spectrum Diversity, which together enhance neuroevolution algorithms' scalability, efficiency, and performance across various problem classes.

Contribution

It proposes a comprehensive neural network representation and a new diversity preservation technique that outperform existing methods like NEAT in neuroevolution tasks.

Findings

Outperforms NEAT on five problem classes

Unified representation captures multiple neural network features

Spectrum Diversity enhances scalability and evolutionary success

Abstract

Learning algorithms are being increasingly adopted in various applications. However, further expansion will require methods that work more automatically. To enable this level of automation, a more powerful solution representation is needed. However, by increasing the representation complexity a second problem arises. The search space becomes huge and therefore an associated scalable and efficient searching algorithm is also required. To solve both problems, first a powerful representation is proposed that unifies most of the neural networks features from the literature into one representation. Secondly, a new diversity preserving method called Spectrum Diversity is created based on the new concept of chromosome spectrum that creates a spectrum out of the characteristics and frequency of alleles in a chromosome. The combination of Spectrum Diversity with a unified neuron representation enables the algorithm to either surpass or equal NeuroEvolution of Augmenting Topologies (NEAT) on all of the five classes of problems tested. Ablation tests justifies the good results, showing the importance of added new features in the unified neuron representation. Part of the success is attributed to the novelty-focused evolution and good scalability with chromosome size provided by Spectrum Diversity. Thus, this study sheds light on a new representation and diversity preserving mechanism that should impact algorithms and applications to come. To download the code please access the following https://github.com/zweifel/Physis-Shard.

Tables

Table 1. TABLE I: Evolvable Parameter Space

Related to	Parameter	Range
Neuron	$adaptationSpeed$	$\{1,7,49\}$
	Type	$\{control,random,sigmoid,threshold,identity\}$
Weight	From Neuron	$\left\{N\right\}$
	To Neuron	$\left\{N\right\}$
	Weight	$\{-1,1\}$
	Neuromodulation	$\{-1,N\}$

Table 2. TABLE II: Parameters for SUNA

Parameter	Value
Number of initial mutations	$200$
Number of step mutations	$5$
Population size	$100$
Maximum Novelty Map population size ($Ma{x}_n$)	$20$
Mutation probability array	$\{0.01,0.01,0.49,0.49\}$
Neuromodulation probability	$0.1$
Control neuron probability	$0.2$
Excitation threshold	$0.0$

Table 3. TABLE III: Parameters for NEAT

Parameter	Value	Parameter	Value
trait_param_mut_prob	$0.5$	trait_mutation_power	$1.0$
linktrait_mut_sig	$1.0$	nodetrait_mut_sig	$0.5$
weigh_mut_power	$2.5$	recur_prob	$0.00$
disjoint_coeff	$1.0$	excess_coeff	$1.0$
mutdiff_coeff	$0.4$	compat_thresh	$3.0$
age_significance	$1.0$	survival_thresh	$0.20$
mutate_only_prob	$0.25$	mutate_random_trait_prob	$0.1$
mutate_link_trait_prob	$0.1$	mutate_node_trait_prob	$0.1$
mutate_link_weights_prob	$0.9$	mutate_toggle_enable_prob	$0.00$
mutate_gene_reenable_prob	$0.000$	mutate_add_node_prob	$0.03$
mutate_add_link_prob	$0.05$	interspecies_mate_rate	$0.001$
mate_multipoint_prob	$0.6$	mate_multipoint_avg_prob	$0.4$
mate_singlepoint_prob	$0.0$	mate_only_prob	$0.2$
recur_only_prob	$0.0$	pop_size	$100$
dropoff_age	$15$	newlink_tries	$20$
print_every	$5$	babies_stolen	$0$
num_runs	$1$

Table 4. TABLE IV: Percentage of improvement (positive) or worsening (negative) of the ablation results in relation to the original one. The problems are Double Pole (DP), Function Approximation (FA), Mountain Car (MA), Multiplexer (MU) and Non Markov Double Pole (NMDP).

Test Type	DP	FA	MC	MU	NMDP
No control neuron	$0\%$	$5.87\%$	$0\%$	$-10.55\%$	$15.76\%$
No linear neuron	$0\%$	$-22.38\%$	$0\%$	$-1.76\%$	$11.18\%$
No neuromodulation	$0\%$	$6.92\%$	$0\%$	$-66.17\%$	$7.88\%$
No random neuron	$-6.28\%$	$7.37\%$	$0\%$	$8.26\%$	$-56.54\%$
No real weights	$-3.29\%$	$9.43\%$	$0\%$	$6.16\%$	$-99.85\%$
No sigmoid neuron	$0\%$	$-3.73\%$	$0\%$	$-3.35\%$	$1.97\%$
No slow neuron	$-6.46\%$	$-32.69\%$	$0\%$	$12.61\%$	$-35.94\%$
No threshold neuron	$-3.13\%$	$6.43\%$	$0\%$	$1.56\%$	$11.77\%$

Table 5. TABLE V: Statistical significance for the ablation tests. Two Mann-Whitney statistical tests were conducted for every ablation test in relation to the control one. One statistical test verify if the results are better than the control one and the other statistical test verify the contrary. W or B means that the ablation has a final result which is respectively worse or better than the control one. Only results significant enough (p-values below 0.05 0.05 0.05 ) are considered. The problems are Double Pole (DP), Function Approximation (FA), Mountain Car (MA), Multiplexer (MU) and Non Markov Double Pole (NMDP).

Test Type	DP	FA	MC	MU	NMDP
No control neuron	$\left(\right)$	$\left(\right)$	$\left(\right)$	W$\left(0.01\right)$	B$\left(0.04\right)$
No linear neuron	$\left(\right)$	W $\left(3.39e-6\right)$	$\left(\right)$	$\left(\right)$	$\left(\right)$
No neuromodulation	$\left(\right)$	B $\left(0.04\right)$	$\left(\right)$	W $\left(2.16e-11\right)$	$\left(\right)$
No random neuron	$\left(\right)$	B $\left(0.03\right)$	$\left(\right)$	B$\left(0.02\right)$	W $\left(8.05e-5\right)$
No real weights	$\left(\right)$	B $\left(0.01\right)$	$\left(\right)$	$\left(\right)$	W $\left(6.78e-12\right)$
No sigmoid neuron	$\left(\right)$	$\left(\right)$	$\left(\right)$	$\left(\right)$	$\left(\right)$
No slow neuron	$\left(\right)$	W $\left(2.87e-6\right)$	$\left(\right)$	B$\left(0.002\right)$	W $\left(0.01\right)$
No threshold neuron	$\left(\right)$	$\left(\right)$	$\left(\right)$	$\left(\right)$	$\left(\right)$

Equations

$$a=f(\sum _{i=1}^n(w_i{x}_i))$$

$$\begin{array}{rl}& In{s}_t=In{s}_{t-1}+\frac{1}{adaptationSpeed}(a-In{s}_{t-1})\\ & y=In{s}_t\end{array}$$

$$\begin{split}&stimulation=\sum_{i=0}^{n}(w_{i}cs_{i})\\ &activation=\left\{\begin{array}[]{cc}true&\mbox{if $stimulation>=threshold$};\\ false&\mbox{otherwise},\end{array}\right.\end{split}$$

$$U=S\setminus \{a_i\}$$

$$uniqueness=mi{n}_{a_k\in U}(dist(a_i,a_k)).$$

$$pos\in (-1.2,0.6)$$

$$v\in (-0.07,0.07)$$

$$a\in (-1,1)$$

$$v_{t+1}=v_t+(a_t)\ast 0.001+\cos (3\ast po{s}_t)\ast (-0.0025)$$

$$po{s}_{t+1}=po{s}_t+v_{t+1},$$

$$y=\frac{x^3}{1000}+0.4x+20\sin (\frac{x}{10})+20\sin (100x),$$

Full text

To download the code please access the following link.

I Introduction

Learning algorithms are the only solution available to problems that are too complex to be solved with hand-coded programs. They are, for example, used to recognize speech in cell phones, aid the trade in stock markets, process and extract knowledge from problems with big data, play games comparatively to humans and so on. Moreover, they are a natural solution to problems that are expected to change over time, since they can learn, i.e. adapt, to changes.

The impact of learning algorithms nowadays is enormous, yet the area has the potential to expand much further. For that, however, it is necessary that learning algorithms be able to work on new problems without the presence of an expert. Current algorithms behave sometimes erratically on new problems or require a lot of trial and error as well as prior knowledge to work.

This motivates us to create algorithms that can learn under problems never seen before and without any prior knowledge. Moreover, this type of investigation have not only a socio-economic value but also a scientific one. Learning can be defined as the ability of improving computational models (representations) to solve problems. It is the idea of starting from zero or little prior knowledge and arriving at a reasonably good solution. Naturally, the more prior knowledge is inserted into a learning algorithm, the less its learning capabilities are important. There is still much to be understood about the learning limitations and inherent trade-offs associated.

To build an algorithm that can face very different problems it is necessary to have a general representation. The most general representation is perhaps a graph with nodes composed of various types of functions. Provided that brains are also graphs of neurons, this justifies the bio-inspiration in neural networks. The only way to optimize not only connection weights but also topology of neural networks is with neuroevolution, because evolution can optimize any kind of problem including ones where the model itself grows or decreases in size. Consequently, neuroevolution is chosen as the underlining basis for this work.

In this article, a neuroevolution based learning algorithm capable of learning various classes of problems without any prior-knowledge is proposed. Moreover, this new neuroevolution based learning algorithm tries to unify most of the neuron representations into one. Since every kind of neuron is specifically good for a certain type of task, the natural selection of the neuron type for each problem suits well the objective of solving a wide range of problems without any prior knowledge. Furthermore, a new niching mechanism and diversity paradigm are created to aid the evolution of this complex representation. The proposed method is called Spectrum-diverse Unified Neuroevolution Architecture (SUNA). In summary, SUNA111Download SUNA’s Code: https://github.com/zweifel/Physis-Shard has the following features:

• •

Unified Neuron Representation - A new neuron representation is proposed that unifies most of the proposed neuron variations into one, giving the method a greater power of representation.

• •

Novelty-Based Diversity Preserving Mechanism - Solutions are not only different, but the frequency and types of neurons and connections used may suggest a completely different approach to solve the problem. For example, the use of many neuromodulated connections against normal connections in a network is a different approach. Such diversity in the approaches used should be preserved and therefore a new novelty-based diversity is proposed which keeps track of previous solutions and preserves the different approaches spotted.

• •

Meaningful Diversity for Large Chromosomes- Instead of comparing chromosomes, this diversity measure compares values of a set of phenotype-based or genotype-based metrics. In other words, spectra222The word spectrum here is used to denote a set of values of histograms or metrics, defining the overall characteristics of the chromosome in a compressed manner. of features are compared. By comparing spectra, the diversity is still meaningful for large chromosomes333High dimensional spaces are exponentially more sparse than low dimensional ones. That is the reason why distance metrics for long chromosomes are less meaningful. To avoid this complication, here the dimensions are converted to a small number of features which compose the spectrum of the chromosome as well as it is problem independent.

• •

State-of-the-Art Results - In all five classes of problems tested, the proposed algorithm had either better or similar results when compared with NEAT.

• •

Few Parameters - SUNA has only eight parameters against $33$ of the NEAT algorithm.

II Neuroevolution

Neuroevolution is an area of research resulting from the combination of the representation power of artificial neural networks [1, 2] with the optimization capabilities of evolutionary methods. For example, instead of using the usual gradient descent methods like backpropagation an evolutionary algorithm is used to train the neural network.

Many works investigated the evolution of fixed-topology neural networks [3, 4, 5, 6]. However, it is hard to specify a good representation in advance. First, the complexity of the representation should match the complexity of the problem, therefore a completely automated algorithm is impossible. Second, although it is possible to represent any function with a single hidden layer, it does not mean that any function is equally easy to represent with a single hidden layer. With the increase of hidden neurons, the search space becomes increasingly huge. That is, the optimization algorithm has to deal not only with plateaus and difficult fitness landscapes, but also with an additional difficulty in the representation because many parameters are present as well as competing conventions (different neural networks that compute the same and exact function).

Deep Neural Networks (DNN) solves the second mentioned problem, i.e., by using many layers it is possible to learn representations in higher levels of abstractions [7],[8]. This justifies why this type of network sets records in image recognition [9],[10] and speech recognition [11],[12] as well as has good results in many other types of applications. Still, in order for DNNs to work specialists are need to set the network parameters and design its topology.

Recently, motivated by the limitations of fixed-topology approaches, there is an increasing interest in algorithms that evolve both the topology as well as the weights. To these algorithms it is given the name of TWEANN (Topology and Weight Evolving Artificial Neural Network). Some of the most famous direct encoding TWEANNs are Cellular Encoding [13], GNARL [14], NEAT [15], EANT [16] and EPNet [17]. Naturally, some of them have also indirect encoding versions while others can be extended to indirect encoding without problems. For a detailed review of TWEANNs please refer to [18].

One of the most successful direct encoding TWEANNs is the NEAT algorithm which has many notable features that made it so popular. We will explain these notable features as well as briefly describe how NEAT works in the following section.

III NEAT

One of the central ideas of NEAT is the innovation number which works as a gene identification and tracking number. Innovation number is an additional value attached to a gene. It enables the recognition of the same gene in different individuals, therefore it is possible to match the same genes while doing crossover. Moreover, it is possible to avoid the problems associated with competing conventions during crossover.

NEAT uses a variable-size genetic encoding where each individual has a list of node genes and a list of connection genes:

• •

Node genes specify if each of their associated nodes is an input, output or hidden node as well as state its innovation number;

• •

Connection genes determine connections between two nodes by specifying the in-node, out-node and weight of the connection. Moreover, there is an associated enable bit (whether or not this gene is expressed) and an innovation number.

There are four types of genetic operators in NEAT: two types of structural mutations, a simple weight mutation and crossover. Structural mutations add complexity to the network by either including a new node in the middle of an existing connection or by adding a new connection between existing nodes (see Figure 1). Additionally, there is the usual weight mutation present in many other methods, where some individuals have the weight value of their connections either perturbed or set to a new random value. Crossover is done by lining up the innovation numbers from both parents in chronological order. Matching genes are inherited while unmatching ones are either inherited randomly when parents are equally fit or inherited from the fittest parent when one parent is fitter than the other. This type of crossover avoids an expensive structural analysis of similarity while still preserving some similar structure present.

Different structures take different time to improve through evolution, e.g., complex structures take longer to improve than simpler ones. Therefore, protecting different structures from unfair competition is necessary if one wants to have an unbiased evolution. NEAT solves this by introducing speciation. Each generation, individuals are separated into species by calculating a distance based on their matching and unmatching genes (i.e., same and different innovation numbers). Individuals that have their distance below a given threshold are from the same species. An explicit fitness sharing [19] is used to make it harder for any species to grow and dominate the entire population. Thus, species have a determined number of offspring slots proportional to the average fitness of its individuals while individuals compete with other individuals within the same species for the allocated offspring slots. The entire population is replaced by its offspring.

NEAT start the search from a relatively small network where all inputs are connected to all outputs without hidden nodes. Notice that the initial population diversity resides in the connection weights which are randomized rather than in the structure. This biases the evolution towards minimal solutions while the genetic operators complexify the structure incrementally by adding more nodes and connections throughout the evolution process.

IV Neural Networks’ Features: A Unifying View

The number of types of neural networks is extremely vast. Although they are called neural networks, they differ in so many aspects such as learning system, architecture444Architecture is defined here as features or dynamics of blocks of neurons such as bottlenecks in the network, connection probabilities, topology dynamics, etc. and model that it is difficult to see them as a unity. Moreover, the features of the neural network used usually reflects the problem it is facing, therefore a unified neural network model is also required to automatically find the solution, independently of problem type.

Motivated by the unification benefits, this section will focus on describing many model features in a unified way. Setting activation functions aside, allow us to divide the model features of a neural network into four broad aspects:

• •

Different neuronal time scales;

• •

Inhibition and excitation of neurons;

• •

Synaptic plasticity;

• •

Feedback;

IV-A Different Neuronal Time Scales

Most neural networks use only one fixed fast time constant, which makes all neurons vary its output as fast as possible in relation to their input. This is important feature in executing tasks which need fast reaction time. However, a slower neuron can take into account a series of past inputs enabling it to decide over long-term events and even store information for other neurons. The importance of time scales have been shown in many articles, for example, in self-localization of a neuroevolved robot in a corridor (the sensory-motor patterns in different time scales are used) [20], in reinforcement learning with multi-models and sub-goals [21], with the Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN) which use memory cells capable of storing/accumulating signals for a very long time (much more than usual RNN) until a forget signal is received [22]. The different neuronal time scales even emerge naturally when a bottleneck is introduced in the neuron topology, creating a layer with long-term task related actions and another layer with short-term reactionary actions [23].

In fact, biological neurons have a membrane time constant that vary widely from neuron to neuron (the time constant depends on the type, density and regulation of the present ion channels) [24]. Therefore, different time scales are present in biological neurons and experiments even show how this temporal hierarchy of neurons relates with the anatomical hierarchy [25]. In other words, the temporal hierarchy is an essential part of the brain.

IV-B Inhibition and Excitation of Neurons

Inhibition and excitation of neurons (the term “neuromodulation of neurons” will also be used) is the ability of some neurons to inhibit (deactivate) or excite (activate) other neurons (see Figure 2). It is widely known that biological neural networks are heavily based on inhibition/excitation principles with groups of neurons influencing how other neurons will behave and if they are going or not going to activate [26], [27] 555In biology, the inhibition/excitation of neurons by other neurons is also called neuromodulation, but the term is rarely used in this sense in the computational field of neuroevolution.. Although not explicitly stated, the dynamics of inhibition and excitation are also present in many machine learning methods such as Decision Trees (DT). If we consider nodes in a DT to be like neurons (in neural trees the DT’s nodes are exactly the same as neurons [28],[29], [30]), instead of processing information through some function and passing to the next connected neurons, they process information and chooses one of the connected neurons to activate. Self-Organizing Map (SOM) is another example that uses excitation and inhibition. SOM makes neurons compete for the input where only one will win and activate [31]. Both SOM and DT usually uses a crisp division, but a fuzzy division can easily be accomplished with fuzzy memberships in a net of excitation/inhibition neurons.

From an information processing perspective, the importance of using inhibition and excitation lies in the use of the divide-and-conquer approach to reduce complexity. It is also possible to create network modules with inhibition and excitation as well as many other complex dynamics that are difficult to be created with other models. Moreover, inhibition/excitation is a completely different model from the usual neuron model used by most neural networks. Therefore, it allows dynamics that are complex to build with the usual neuron model to be easily built.

IV-C Synaptic plasticity

Synaptic plasticity has been motivated largely by its biological foundations, but few is known about its useful information processing properties. There are homosynaptic and heterosynaptic mechanisms [32]:

• •

Homosynaptic - weight update is only a function of the pre and post synaptic neurons (e.g., Hebbian learning);

• •

Heterosynaptic - the synaptic is neuromodulated, i.e., the same as homosynaptic with the addition of a modulatory neuron that influences strongly the Hebbian plasticity (e.g., modulated Hebbian learning).

In homosynaptic mechanisms, researchers reported that the use of synaptic plasticity aided their methods to adapt to environment changes [33] but depending on the task it may not give any improvements [34]. It seems that the change of weights based on the relationship between the connected neurons (e.g., Hebbian learning) creates certain complex dynamical systems as a result, but there is no reason why the same dynamical systems could not be created with a given structure and some neurons, i.e., without changing weights online.

Having said that, heterosynaptic mechanisms employ neuromodulation666Neuromodulation, in artificial neural network, is the modulation of connection weight(s) by another neuron. of the connections and differ substantially from homosynaptic mechanisms. The results obtained from using such mechanisms are much more encouraging [35], [36], [37], [38], [39]. From an information processing point of view, the neuromodulation allows for another neuron to influence the relationship between two connected neurons. Recall that the relationship between neurons coded by the connection is never dependent on another neuron. Therefore, the heterosynaptic mechanism expands arguably the representation capabilities of the computation model. This work will use only heterosynaptic mechanisms. Figure 3 shows a simple example.

IV-D Feedback

Feedback is one of the ways that a network has to keep information from previous runs (e.g., one way of codifying memory). This is the reason why neural networks with feedback can solve non-Markov problems. Most importantly it is the only way that neurons in a processing sequence may have access to the resulting outputs from subsequent neurons (see Figure 4 for examples of network structures with feedback). This property is extensively used in control systems, electronic engineering, biology, social sciences and many other areas.

V Unified Neural Model

In this section, the proposed unified neural model will be described.

V-A The Extended Neuron

Neural models are built from the interconnections and features of simple units, the neurons. The neuron used here is an extended version of the widely used neuron model employed by the perceptron, multi-layer perceptron, and most other neural network algorithms (will be referred to as ”usual neuron model” in this paper). This extension is done in two ways:

• •

Adaptation Speed - Every neuron has its own reacting speed, that is, some neurons may take longer to change their outputs;

• •

Type - Neurons have various activation functions.

Figure 5 shows a diagram of the behavior of a single neuron. Mathematically speaking, it can be determined by the following equations:

[TABLE]

[TABLE]

Equation 1 is basically the same as the usual neuron model, where every input $x_{i}$ is multiplied by its associated weight $w_{i}$ for $n$ inputs. The result $a$ from applying the activation function $f()$ is, however, not used to update the output $y$ directly. It is used instead in a delta rule like equation (Equation 2) that takes the neuron adaptation speed $adaptationSpeed$ as parameter to update its previous internal state $Ins_{t-1}$. The reasoning behind this equation is that if the neuron input changes, the neuron output would change completely after $adaptationSpeed$ iterations, i.e. creating a slower neuron. Finally, the current internal state $Ins_{t}$ is used as output.

The reason behind these modifications go beyond bio-inspiration. Different adaptation speeds result in a multi-scale network that is capable of taking long-term changes into account. For example, finding an approximation to the average of a variable is as simple as connecting a slower neuron to it. Moreover, the use of a single type of neuron (only one activation function) is an unnecessary bias. The use of some types of neurons allows for the neural network to fit different problems without increasing much the searching space.

V-B Control Neuron and Control Signals

One of the motivations behind TWEANNs is to evolve problem-related structures automatically. In fact, sometimes a simple structure that switches between different behaviors (set of neurons) is enough to solve a task [40]. However, there is no way to turn off a set of neurons in current TWEANNs so a question is raised. How can structures such as a basic switch system come to life?

We propose to solve this problem by adding a different signal type called control signal. Control signals will be solely responsible for activating and deactivating neurons, i.e. they will not interfere with their inputs. Therefore, the following equation is added to decide whether the extended neuron activates or not:

[TABLE]

where $cs_{i}$ is the control signal and $w_{i}$ its relative weight for $n$ control signals. Therefore, the neuron activates if the $stimulation$ is more or equal to the parameter $threshold$, because the $activation$ is set to $true$. Otherwise, the neuron does not activate. Here, the $threshold$ is fixed for all neurons.

To make all these possible, control signals need to be somehow present. This makes necessary the introduction of an special neuron called control neuron. The control neuron is exactly like all other neurons aside from the fact that it does not output normal signals but control signals. All control neurons have threshold activation functions.

V-C The Genome

To evolve neural networks it is necessary to define a genome. The genome used is a direct encoding scheme made by a list of connections and a list of neurons (see Figure 6).

The evolvable parameters and their respective range values are shown in Table I, where $N$ is the set of neurons present in the respective chromosome. Notice that the type parameter values input identity (input neuron with identity activation function) and output identity (output neuron with identity activation function) (see Figure 6) as well as the whole interface index parameters are not part of the evolvable parameters although they are present inside the genome. This happens because they are only created by the initializing step of the evolution and kept fixed afterwards.

V-D Order of Neuron Execution

To let the control neurons decide which neurons are going to execute, the control neurons need to be executed first. Afterwards, control signals are established and the remaining neurons are executed. In other words, the following order is respected:

• •

Input neurons;

• •

Control neurons that do not receive connections from other control neurons;

• •

Remaining control neurons that are activated;

• •

Remaining neurons that are activated and have an absolute output higher than $0.001$;

Input and output are always cleared after the network execution even when the respective neurons were not activated. That is, if a given input or output neuron was not activated, the respective input or output is set to zero.

VI Evolving the Unified Neural Model

To evolve structures, special evolutionary methods are needed. A new evolutionary method is proposed that is based on a new diversity paradigm. Here, we show how diversity of structures can be achieved without creating a dependence on the problem.

An overview of the evolution process is shown in Figure 7. The steps will be described in detail in the following subsections.

VI-A Initialization

First, all chromosomes in the population are initialized to be a set of input and output neurons, one for each input/output present in the problem. For these neurons the interface index is set to be the index of the respective input/output. To create an initial diversity, some initial mutations are applied for all chromosomes.

VI-B Evaluation

In the evaluation stage every chromosome present in the population is activated for a whole trial. At the end of each trial, the average reward received throughout is used as fitness. This process is repeated until the entire population has been activated. Afterwards, the spectrum of every chromosome is calculated.

VI-C Spectrum Diversity

Diversity and protection of innovation is a challenging problem for TWEANNs, basically, because finding a metric to compare the structures of TWEANNs is difficult. Speciation tries to approximate this comparison by using innovation numbers (a certain type of id) on the genes and therefore the alignment of such genes gives a fast approximation of the structure similarities [15]. However, two structures originated from different evolutionary paths are always seen as totally different and unmatching structures because they differ in innovation numbers. This happens even if they are the same or similar in structure. And since genes are compared one per one it does not scale well with the size of the chromosome.

In Physics and Chemistry, the opposite problem takes place. Atoms combine with each other into complex structures, but it is discovering these structures which is a hurdle. To solve these problems various types of spectra are used (e.g., nuclear magnetic resonance spectrum).

Similarly, here we propose to use spectra to identify and compare structures. The spectrum is itself designed for the representation though, composed of a set of genotype’s and phenotype’s properties. Computationally speaking, the spectrum is an array with each property being an element of this array. Here the spectrum is composed of the following properties:

• •

The number of identity neurons;

• •

The number of sigmoid neurons;

• •

The number of threshold neurons;

• •

The number of random neurons;

• •

The number of control neurons;

• •

The number of slower neurons (adaptation speed greater than one).

This combination of properties divides the set of solutions by how they approach to solve the problem.

Once the spectrum of all chromosomes are calculated, their spectrum are fed into the Novelty Map [41] (Section VI-D) where only the spectra that are mapped into the same cell creates a group (species) and compete. The entire process of calculating the spectra and insertion in the Novelty Map is called Spectrum Diversity, because the spectrum of the chromosomes is used to separate the population into groups(species), i.e., diversity is kept by creating species based on their spectrum similarity.

Spectrum diversity uses a measure of diversity based on the distance between the spectra of two structures. The advantages of using this type of diversity are:

• •

Problem Independence - The metric is not defined based on the problem characteristics, therefore it works independent of problem type and size;

• •

Scalability - Since the spectrum should describe properties of the structure as a whole, the size of the chromosome does not interfere with the metric.

Notice that measures from complex networks and graph theory can be also be used, joining the discipline of complex networks with TWEANNs. Actually, by using metrics from the complex networks field it is possible to achieve many types of structure diversity, for example, degree distribution diversity, assortativity diversity, etc.

VI-D Novelty Map

Novelty Map is a table with the most novel inputs according to a given novelty metric (see Figure 15). When an input is presented to the map, competition takes place where the cell with the closest weight array wins. This winner cell is activated and can be used in many ways, here the winner cell is used to identify to which species (group) the input pertains. Afterwards, the table is updated by substituting the weight array of the least novel cell (according to the novelty measure) with the input array if and only if the input array has higher novelty.

This table shares some similarities with the self-organizing map [42] and the neural gas [43], but its behavior is independent of input frequency and it uses fewer cells to map the same input space (cell’s efficiency).

The novelty metric used in this article is the uniqueness. Let $S$ be a set of arrays. Uniqueness is defined for an array $a_{i}$ in relation to the other arrays in $S$ by:

[TABLE]

VI-D1 Novelty Map Population

Novelty Map Population is a Novelty Map where cells are subpopulations and inputs are some value related to the individual. Here, the spectrum of chromosomes are used as input.

VI-E Selection

In the selection stage, only one individual survives inside every cell (species) of the Novelty Map. To decide which individual is going to survive, the individuals are compared based on their fitness. And in the case of draw, the individual that uses less neurons wins (i.e. the least complex wins).

In this sense, subpopulations are the same as niches, because chromosomes that fall inside the same subpopulation have a similar histogram and therefore are similar themselves.

After selection, the individuals that survived become parents and the algorithm enters in reproduction stage. As a matter of fact, to keep the number of parents constant, if a subpopulation is empty, a random valid parent from a different subpopulation is chosen and copied instead.

VI-F Reproduction

With the parents defined, the remaining vacancies in the population are filled by the following sequence of procedures:

A parent is chosen at random; 2. 2.

Some step mutations are applied to the parent to create a child; 3. 3.

All connections have a $50\%$ chance of being perturbed with a random value in the range $[-w,w]$, where $w$ is the current weight of the respective connection.

A mutation can be any of the following procedures:

• •

Add a neuron. Everything is randomly chosen, but first the neuron has an additional probability of being a control neuron. Additionally, two new connections are created connecting this neuron to the network (one random connection from this neuron and another random one to this neuron);

• •

Delete a neuron (all connection to the respective neuron are also deleted). Input/output neurons cannot be deleted;

• •

Add a connection. There is a probability of the connection be neuromodulated (neuromodulation probability). Everything else is randomly decided;

• •

Delete a connection.

Any of the above mutation procedures have a chance of occurring defined by the mutation probability array. The mutation probability array is composed of four real numbers relative to the probability of respectively adding a neuron, delete a neuron, add a connection and delete a connection. So for example $\{0.1,0.1,0.4,0.4\}$ would mean $10\%$ chance of adding a neuron, $10\%$ chance of deleting a neuron, $40\%$ chance of adding a connection and $40\%$ chance of deleting a connection.

VII Experiments

In this section both SUNA and NEAT will be tested on five different classes of problems, each one requiring different learning features to solve. The set of parameters are kept the same for all the problems for both algorithms (see Section VII-A).

VII-A Experiments’ Settings

The NEAT code used is the 1.2.1 version of the NEAT C++ software package [44]. Table III shows the NEAT parameters. Actually, the used parameters are the same as the one provided with the package for solving double pole balancing task. A couple of variations of it were tested, but the original one performed better. Moreover, some modifications were necessary to make NEAT work on problems with negative fitness, since it does not work out of the box. Therefore, a value big enough ($2000$) was always added to the final accumulated fitness, transforming the negative fitness into a positive one. Additionally, to satisfy NEAT’s requirements, for all problems the input range was converted to $(-1,1)$ while the output range was transformed into $(0,1)$. This conversion naturally informs the algorithm about the maximum/minimum of both input and output, simplifying the problems at hand. Such changes are not necessary for the proposed method.

Ablation tests are also conducted to evaluate the importance of most of its features as well as understand when they are important. These tests are conducted over problems without any transformation of input/output. In other words, the algorithm does not know the range of input/output, verifying further its learning capabilities.

All results are averaged over $30$ runs777Run is defined as the sequence of trials until the maximum number of trials is reached. and only the best result among $100$ trials888Trial is a set of iterations between agent and environment until a problem defined stopping criteria is met. is plotted. Every run, when not specified otherwise, has a maximum of $2\times 10^{5}$ trials.

VII-B Mountain Car

The mountain car problem [45] is defined by the following equations (Figure 9 shows an illustration of the problem):

[TABLE]

where $pos$ is the position of the car, $a$ is the car’s action and $v$ is the velocity of the car. The starting velocity and position are respectively $0.0$ and $-0.5$. If $v<0$ and $pos\leq-1.2$, the velocity is set to zero. When the car reaches $pos\geq 0.6$ the trial is terminated and the algorithm receives [math] as reward. In all other positions the algorithm receives $-1$ as a reward. Moreover, if the algorithm’s steps exceed $10^{3}$ the trial is terminated and the common reward of $-1$ is returned to the algorithm.

NEAT can only deal with certain ranges of input/output, therefore the input was converted to the $(-1,1)$ range while the output was converted to the $(0,1)$ range. Naturally, this conversion informs the algorithm about the maximum/minimum of both input and output, an information that was not readily available in the original problem, therefore the problem is to some extent simplified.

Figure 10 shows a comparison between SUNA and NEAT in the mountain car problem. SUNA performs better than NEAT, converging as fast as NEAT even though SUNA sees a bigger search space. The reason behind the worse performance in NEAT seems to be related with unstable individuals taking over the population from time to time, making the results vary around the best solution but rarely staying in that position. Moreover, procedures that check for stagnation make things worse when NEAT has already reached a good result.

The non-simplified version of the problem, i.e., with raw input and output is shown in Figure 11. SUNA performs equally well on all problems, showing that the presence or absence of normalization in the input/output is not an issue. NEAT, on the contrary, struggles with raw input and when the raw output is present it performs very poorly. It is understandable, NEAT has only sigmoid input nodes that squeeze values into the $(-1,1)$ range and outputs nodes in the $(0,1)$ range and therefore can not cope with this type of problem. Having said that, it is still a limitation, compromising the automatization and limiting the application to some problems.

VII-C Double Pole Balancing

Double pole balancing is the famous problem of balancing a pole above a cart by only moving the cart forward or backward (see Figure 12). This problem has six observing variables. The dynamics used here are the same mentioned by the following papers [4],[46],[15]. In fact, the code was adapted from [44] which is the same code used in other papers [46],[15]. All the original parameters of the problem were also kept the same.

Figure 13 shows that SUNA and NEAT converges to the same final performance. In other words, both algorithms learned to balance the pole for $10^{5}$ steps (the maximum allowed) in all the tests. SUNA converges slower than NEAT. This is expected due to the greater complexity of the SUNA’s model.

VII-D Non-Markov Double Pole Balancing

Non-Markov double pole balancing has the same dynamics as the double pole balancing. However, the agent only observes three variables (position of the cart and angles of both poles) instead of six. To keep the poles balanced it is necessary to estimate the velocity of the cart and pole angles. Therefore, this problem presents an additional difficulty. The previous input needs to be somehow kept and used to compose an estimate of velocity.

Notice that in this article the double pole balancing fitness function is used, i.e., a modified fitness function that penalizes oscillations is not used [15]. Thus, this formulation of non-Markov double pole balancing problem is relatively more difficult.

Comparison results are shown in Figure 14 SUNA performs much better than NEAT in this problem. In fact, SUNA achieves $70\%$ of the maximum possible average accumulated reward while NEAT reaches only $37\%$. But what features present in the proposed algorithm enable it to outperform NEAT? This question is answered in Section VII-G with ablation tests. However, there are basically three important features: slow neurons (responsible for computing approximation of derivatives, averages and so on), real weights (accurate computation of estimates) and random neurons (exploration).

VII-E Multiplexer

Multiplexer is a binary problem with a single bit output and with the input composed of data and address variables. To compute the correct output, $abits$ address variables are used to select one of the other $2^{abits}$ data variables. This selected data variable is the correct output. In this manner, the algorithm must first separate the address input from the data input and then use the address information to select the data part of the input. This problem is coded as a reinforcement learning problem where each correct answer is rewarded with [math] and each incorrect answer is rewarded with $-1$. Every possible set of inputs are presented during a trial in random order, not allowing learning algorithms to memorize the sequence of outputs instead of the multiplexer function.

Figure 15 shows that SUNA outperforms NEAT in the multiplexer problem with $abits=3$ (i.e., three address variables and eight data variables). After $2.0\times 10^{5}$ trials, an average of $-620$ ($\approx 70\%$ accuracy) accumulated reward is achieved by SUNA while NEAT reaches $-720$ ($\approx 65\%$ accuracy). However, SUNA continues improving its accuracy further and if more trials are allowed it would reach much higher accuracy. NEAT, on the other hand, reaches rapidly its peak of accuracy without further improvements. Ablation tests done in Section VII-G show that one of the features that allows SUNA to outperform NEAT is neuromodulation. See Section VII-G for details.

VII-F Function Approximation

In this problem the algorithm has to learn a sequence given by the following equation:

[TABLE]

where $x$ and $y$ are respectively the input and output of the problem. Moreover, the value of $x$ varies from $-100$ to $100$ in unit time steps. The reward is the negative modulus of the difference between the agent’s action (agent’s estimation of $y$) and the correct sequence output $y$.

This problem cannot be solved properly by NEAT, since its input and output need to be real numbers of unknown range otherwise the problem is unreasonably easy. Therefore, here only the SUNA’s result is plotted (see Figure 16). SUNA gets a very close approximation to the curve, Section VII-G will expose the importance of slow neurons and linear neurons in these classes of problems.

VII-G Ablation Tests

In contrast to the previous tests, here all the tests do not use any pre-processing system such as the conversion of input/output to a certain range. So all input/output conversion necessary for using NEAT was excluded. A natural consequence is that previous problems get more difficult.To enable enough time for the algorithm to converge in these more difficult problems, runs have a maximum of $10^{6}$trials instead of $2\times 10^{5}$ used in previous tests. These types of tests were preferred to stress SUNA and verify its capabilities.

The ablation tests are composed of nine tests. A control test and eight ablation ones. Table IV shows the percentage in improvement or worsening after the ablation and Table V shows the statistical significance of the tests. The following subsections will discuss every ablation in detail.

VII-G1 No Linear Neurons

Linear neurons are specially important in problems with real input/output and input/output with wide range. The ablation test on function approximation confirms this affirmation, since the results degrade strongly when linear neurons are removed. In fact, this result is statistically significant.

VII-G2 No Control Neurons

As explained before in Section IV-B, with the introduction of control neurons it is now possible to create systems that switch on/off behaviors. So the representation capabilities improved strongly but the tests with the ablation of control neurons show mixed results. Consequently, control neurons does not seem to be used to their maximum representation potential by the current evolutionary algorithm.

There is a reason for the above. Since certain groups of neurons work together forming functions, representation-wise, a higher level of control can be achieved if groups of neurons are activated or deactivated. However, if the connections do not perfectly connect to all neurons involved in the function, instead of adding a higher level of control, the current level of control is disrupted and control neurons become instead another piece inside the monolithic block of neurons. In other words, control neurons which are unwisely connected to the network, do not switch on/off behaviors (set of neurons) but just neurons.

VII-G3 No Neuromodulation

The ablation tests show that neuromodulation is of utmost importance (a statistically significant $66\%$ worsening if removed) for the multiplexer problem. Actually, it is easy to see why neuromodulation is important in a problem like multiplexer that has input selection as its main task, because neuromodulation does exactly that, it modulates a node based on another one.

VII-G4 No Random Neurons

Random neurons may seem at first glance somewhat weird, summing with the fact that they are of extreme importance for the non-Markov double pole problem is yet more perplexing. However, notice that random neurons are exactly important on problems that require some form of exploration of the environment (e.g., non-Markov double pole) while being irrelevant where no exploration is necessary (function approximation and multiplexer).

It is surprising that even without any kind of exploration dynamics hard-coded, that the algorithm develop by itself, at least initially, dynamics that explore the environment.

VII-G5 No Real Weights

This time, tests verify the importance of using real weights, i.e., these ablation tests use only -1, 1, or neuromodulation (another neuron’s output) as possible weight values. The results show that setting the non-Markov double pole balancing aside, in all problems the ablated algorithm’s performance is similar to when real weights are used, with the double pole balancing problem having improved a bit the results after the removal of real weights. However, for the non-Markov double pole balancing problem the removal of real weights causes substantial decrease in performance.

This can be explained by the following reasoning. The absence of real weights is a constraint added to the problem and when this constraint helps to find a good solution (in the case of double pole balancing) it is surely welcome, but sometimes it makes impossible to find good solutions (non-Markov double pole balancing). Notice that the only difference between these two problems (non-Markov double pole and double pole) is the presence/absence of the three velocity variables and therefore when they are not present it is necessary to construct them with recurrence and some calculation, the recurrence is present but it seems that this calculation is never precise enough without real weights and therefore the algorithm is never able to solve the non-Markov version.

VII-G6 No Sigmoid Neurons

The removal of sigmoid neurons is shown to have no impact on the system performance. All the comparisons had a small percentage difference as well as were not statistically significant. This may be related with the similarity between sigmoid and threshold neurons. Sigmoid neurons have a non modifiable steep curve given by a hyperbolic tangent which has the same output for almost all inputs, aside from small portions of the input range near zero. Therefore, the system does not suffer from losing one or the other.

VII-G7 No Slow Neurons

Slow neurons are important when the information needs to be preserved from previous iterations. That is why they are very important to non-Markov double pole and function approximation problems. Notice that slow neurons allow for the computation of an average of its input (i.e., the input gets accumulated and is naturally averaged in its output), therefore they are good estimates of rates of variance as well as derivatives.

VII-G8 No Threshold Neurons

As mentioned before, threshold neurons and sigmoid neurons have similar activation functions. So removal of one or the other does not impact on the performance. This justifies not statistically significant results as well as small percentage difference.

VIII The Laws of Learning

It was shown that each problem class requires some features from learning algorithm in order to be tackled. However, these features are not specific to each problem class. Many problem classes require similar learning features. Therefore, the question of whether there is a minimal set of features required to solve all problems is raised.

Thus, similarly to the laws of Physics, which are the minimum set of dynamics that describe the universe. Are there a minimum set of learning features (neuron types, recurrency, etc…) that would allow a learning algorithm to learn any problem?

We think there is a minimum set. In this work we identified some important learning features. But there are certainly many more that need to be identified and unified.

IX Extensions to NEAT and Extensions to SUNA

NEAT has various extensions, such as FS-NEAT [47] where the initial topology is set to only one input connecting to the output instead of the initial fully connected topology between inputs and outputs, RBF-NEAT [48] which uses an additional mutation that adds Radial Basis Function (RBF) nodes, Cascade-NEAT [49] where only the connections to the last added node are evolved and SNAP-NEAT [49] which integrates mutations of three algorithms (NEAT, RBF-NEAT and Cascade-NEAT) into one.

Regarding indirect encoding, Compositional Pattern Producing Networks (CPPN) is an abstraction of natural development that uses neural networks to create an indirect encoding instead of developmental encodings. Usually, CPPNs are developed with the algorithm called CPPN-NEAT that uses NEAT to evolve it [50]. Moreover, HyperNEAT is an indirect encoding based algorithm which uses CPPN-NEAT to create patterns of weights in substrates [51].

In fact, many of the NEAT extensions can be easily applied to SUNA. For example:

• •

Similarly to RBF-NEAT, RBF nodes can be included in SUNA;

• •

Connections to old nodes can be frozen, having only the connections to recent added nodes evolve (similar to Cascade-NEAT);

• •

Additional mutation operators can be used or mixed, integrating other approaches (SNAP-NEAT);

• •

CPPNs can be developed using SUNA in the same way that they were developed with NEAT, using a CPPN-SUNA algorithm;

• •

HyperSUNA can be developed using either a CPPN-SUNA or by substituting CPPN with SUNA entirely.

X Conclusions

This paper proposed a new TWEANN that joins most of the neural network features into one unified representation. In fact, a natural consequence of the unified representation is a much more powerful representation power. This affirmation was verified by the tests where the proposed algorithm outperformed NEAT on most of the problems.Moreover, ablation tests demonstrated that good results can only be obtained with the presence of the added new features. Therefore, this work give steps forward an algorithm that can learn any problem and we raise the question of if there is a minimal set of features that can let an algorithm learn anything.

Actually, representation alone is not enough, a diversity preserving method that called Spectrum Diversity is also necessary. Spectrum Diversity allows novel enough chromosomes to be kept even when their fitness is abysmally poor. Consequently, this allows different approaches to develop without the deleterious competition of faster evolving ones. Moreover, Spectrum diversity scales better with the size of the chromosome than Speciation because of its use of chromosome spectra instead of the genes themselves. This diversity method together with the chromosome spectrum concept should find uses in many other algorithms, specially the ones that utilize a great variety of gene types in their genotype as well as applications where the chromosomes are high dimensional.

Thus, the current work sheds light on a new representation as well as a new diversity keeping method and its associated spectrum concept, opening up arguably better possibilities for the creation of algorithms. Moreover, the overall good results motivates further applications of the approach to real-world problems and extensions such as an indirect encoding version.