Scalable Co-Optimization of Morphology and Control in Embodied Machines

TL;DR

This paper introduces a technique called morphological innovation protection that improves the co-optimization of robot body plans and control policies by temporarily reducing selection pressure on recent morphological changes, helping avoid local optima.

Contribution

The paper presents a novel method for co-optimizing morphology and control in embodied robots, inspired by embodied cognition, to enhance evolutionary search and avoid stagnation.

Findings

Method enables convergence to highly fit morphologies from diverse initial conditions.

Technique sustains fitness improvements further into the optimization process.

Approach helps avoid local optima in evolutionary robot design.

Abstract

Evolution sculpts both the body plans and nervous systems of agents together over time. In contrast, in AI and robotics, a robot's body plan is usually designed by hand, and control policies are then optimized for that fixed design. The task of simultaneously co-optimizing the morphology and controller of an embodied robot has remained a challenge. In psychology, the theory of embodied cognition posits that behavior arises from a close coupling between body plan and sensorimotor control, which suggests why co-optimizing these two subsystems is so difficult: most evolutionary changes to morphology tend to adversely impact sensorimotor control, leading to an overall decrease in behavioral performance. Here, we further examine this hypothesis and demonstrate a technique for "morphological innovation protection", which temporarily reduces selection pressure on recently morphologically-changed individuals, thus enabling evolution some time to "readapt" to the new morphology with subsequent control policy mutations. We show the potential for this method to avoid local optima and converge to similar highly fit morphologies across widely varying initial conditions, while sustaining fitness improvements further into optimization. While this technique is admittedly only the first of many steps that must be taken to achieve scalable optimization of embodied machines, we hope that theoretical insight into the cause of evolutionary stagnation in current methods will help to enable the automation of robot design and behavioral training -- while simultaneously providing a testbed to investigate the theory of embodied cognition.

Full text

I Introduction

Designing agile, autonomous machines has been a long-standing challenge in the field of robotics [31]. Animals, including humans, have served as examples of inspiration for many researchers, who meticulously and painstakingly attempt to reverse engineer the biological organisms that navigate even the most dynamic, rugged, and unpredictable environments with relative ease [11, 32, 38]. However, another competing approach is the use of evolutionary algorithms to search for robotic designs and behaviors without presupposing what those designs and behaviors may be. These methods often take inspiration from the evolutionary method itself, rather than the exact specifications of any given organism produced by it.

The use of an evolutionary algorithm for automated design comes with many benefits: It removes the costly endeavour of determining which traits of a given organism are specific to its biological niche, and which are useful design features that can provide the same beneficial functions if instantiated in a machine. It can yield machines that do not resemble any animals currently found on earth [23], as it allows for machines that are specialized for behaviors and environments that differ from those of the model organism. Additionally, the optimization process can serve as a controlled and repeatable test-bed for the study of evolutionary, developmental, or behavioral theory [25, 26, 4].

However, the generalization of design automation to include both the optimization of robot neural controllers and body plans has proven to be problematic. While recent successes have demonstrated the potential of effective optimization for the control policies of agents with fixed morphologies [17, 13, 27] or – to a lesser extent – the optimization of morphologies (body plans) for agents with minimal and fixed control policies [12, 10, 1], the co-optimization of the two has seen very limited success [16, 8].

This inability to perform robot brain-body co-optimization at scale has been experienced and noted informally by many researchers involved with robot optimization, yet published rarely. Thus the lack of publication is presumably because the field lacks incentives for the publication of negative results [14], rather than a lack of negative results in unpublished works. Joachimczak et al. provide an anecdotal example of premature convergence in the co-optimization of robot brain and body plan (Fig. 19 of [22]). Cheney et al. [8] further analyze the phenomenon of premature convergence in embodied machines and suggest that traditional evolutionary algorithms are hindered in this setting primarily in their ability to perform continued optimization on the morphology of the robot. They hypothesized that the premature convergence may be due to an effect of embodied cognition, in which an individual’s body plan and brain have an incentive to specialize their behaviors to complement one another. This specialization makes improvements to either subsystem difficult without complementary changes in the other (a highly unlikely event given current algorithms) and thus results in an embodied agent which is fragile with respect to design perturbations.

II Background

Attempts to solve this problem of fully-automated robot evolution are frequently traced back to the work of Sims [34]. This work introduced the use of evolutionary algorithms to produce goal-directed behaviors and morphologies simultaneously. Despite the advance this work represented, the evolved robots tended to be composed of a small number of components (figures show a mean of 6.042 segments per robot, with each typically controlled by a few neurons). It has been suggested that current computational power should have vastly increased the scale and complexity of robots evolved using Sims’ and similar methods. Yet surveys of evolved robots (e.g. [16, 5]) fail to exhibit any significant increase in size or complexity.

A wide range of hypotheses for the lack of scalability have been proposed. Some focused on a lack of efficient evolutionary search algorithms [20, 24, 29] or genetic encodings [21, 6, 10], while others pointed to a lack of incentives for complexity in the simple tasks and environments [1, 7]. Yet attempts to evolve robots using methods designed to overcome these challenges have yet to obviously surpass Sims’ work in terms of complexity and scale. This work investigates a different hypotheses, first suggested in [8], that considers the way in which an agent’s brain and body plan interact during the optimization process.

III Methods

III-A The Interdependency of Body and Brain

We investigate the notion that the specialization of brain and body plan to one another during evolution creates a fragile co-dependent system that is not easily amenable to change. This specialization creates local optima in the search space and premature convergence to suboptimal designs. In this paper, we explore a direct solution to the problem of fragile coupled systems: explicitly readapting one subsystem (e.g. the body plan or the brain) after each evolutionary perturbation to the other. The proposed method differs from a traditional evolutionary algorithm, which evaluates the fitness of a newly proposed variation immediately (i.e. with no readaptation), and uses only this valuation of fitness to determine the long term potential of that variation.

As a thought experiment, consider a hypothetical robot with a partially optimized body plan (for example a quadrupedal form) and a partially optimized controller (for example the legs swing forward and back through the sagittal plane). Suppose that this controller has co-adapted to the morphology during evolution such that for each step forward, the robot contracts a muscle near its hip with enough force to swing one of its legs forward with just enough force to land in front of the body and successfully take a step. Consider a variation of this morphology proposed by an evolutionary algorithm in which the new robot possesses longer legs but the controller has not changed.

If this machine were being evolved for rapid locomotion, having longer legs and being able to take longer strides would be beneficial and this variation should be more successful than the original design. However, during evaluation of this new robot, the original controller applies the same amount of force to the now-longer leg, failing to move it, and thus frustrating the robot’s ability to walk in a coordinated fashion. Current evolutionary methods would treat this robot as the recipient of a detrimental mutation and remove it from the population. If all such variations display an immediate detrimental fitness impact and are rejected (regardless of their long term potential), then the evolutionary algorithm has converged to a local optima in the search space.

However, the newly proposed morphology would have resulted in a robot which outperformed its predecessor, if coupled with a controller suited to that morphology. We can determine that the newly proposed morphology is superior by suppressing mutations to that body plan and allowing readaptation of its controller to properly coordinate behavior.

Herein lies the foundation for our proposed algorithm: readapt each controller until the new proposed morphology is more fit than its predecessor.

III-B Controller Readaptation

The most obvious method for modeling controller readaptation would be to protect any lineage that has recently experienced a mutation to the body plan by allowing it to undergo several generations of evolutionary change restricted to the control subsystem. If any member of the lineage achieves higher fitness than the pre-mutation ancestor during that time period, the descendant is retained. Otherwise, the new morphological variant dies out.

However, it is unclear how to set the time period for this protection a priori. Surely the amount of time a controller takes to readapt to a new morphology depends on many specific features of the complexity, genetic encoding, desired behavior, and current ability level of the robot (which changes over optimization time). Determining the correct value of this parameter would require a full parameter sweep over various values of readaptation time for each new combination of brain, body, and environment. If our goal is simply to optimize a robot, then searching for this value in each unique optimization scenario is intractable.

III-C Proposed Method: Morphological Innovation Protection

In response to the unintuitive nature of the optimal value for readaptation length, our proposed approach is free of this parameter. Descendants of robots that experience morphological mutations are retained in the population and the number of generations that have elapsed since that mutation occurred are tracked (referred to as the “morphological age” of the robot). If two individuals are found in the population such that the latter robot exhibits better performance on the desired task and has experienced fewer generations since a morphological mutation than the former robot, this latter robot is said to “dominate” the former robot in this mutli-objective optimization (on “age” and “fitness”) – and the former robot is removed from the population. In effect, the latter robot has exhibited an ability to not only recover from its ancestor’s morphological mutation but improve upon it (and others in the population). The concept of tracking age using it as an optimization objective are borrowed from [20, 33]. The major difference here being that the age refers to the length of time that a subsystem of the agent (e.g. the morphology) has remained unvaried, rather than (the original definition of) the total time since a random individual was introduced to the population.

This procedure has the effect of “protecting” new morphologies with poorly adapted controllers, and thus we will henceforth refer to this procedure as “morphological innovation protection.” This protection is a form a diversity maintenance, though reduced selection pressure for newly mutated morphologies. Various other methods exist for encouraging diversity (e.g. fitness sharing, crowding, random restart parallel hillclimbers [18], novelty [24], or speciation [36]), however age was chosen for its simplicity of implementation, its parallels to multi-timescale learning in biology, and because it helps to avoid the cost of extended control re-optimization for non-promising morphologies – since the the age-pareto optimization allows fitness comparisons between all new “child” morphologies that have had equal readaptation time, even if they are not yet fully readapted (rather than making comparisons only after a predefined amount of readaptation).

III-D Evolutionary Algorithm

All optimization is performed by a population-based evolutionary algorithm. All trials follow a $(\mu,\lambda)$-Evolutionary Strategy [3] with $\mu=25$ parents and $\lambda=25$ mutants for a population size of 50. Trials last for 5000 generations. Crossover was not considered in this work. Mutation had a 50% chance of creating a variation to either the morphology or the controller of a given robot, but not both. Other ratios of morphology:controller mutations were considered (1:99, 20:80, 50:50, 80:20, and 99:1), but none showed a significant effect on resolving the premature convergence and resulting fitness in preliminary trials without innovation protection.

III-E Genetic Encoding for Soft Robot Morphologies

Consistent with prior work studying the co-optimization of robot morphologies [10] and controllers [8], we choose soft robots as our model system due to the open-ended complexity of deformable voxel-based morphologies and distributed controllers. The soft robot morphologies are encoded with a Compositional Pattern Producing Network (CPPN) [35]. The CPPN encoding produces the cell fate of each voxel in the robot through a type of neural network that takes each cell’s geometric location ($x,y,z$ Cartesian coordinates and $r$ radial polar coordinate) and outputs a variety of “morphogens” (in this work, there is one to determine whether a cell is present in that location and one to determine whether a present voxel should be a muscle or a passive tissue cell). Since nearby voxels tend to have similar coordinate inputs, they also tend to produce similar outputs from the network – creating continuous muscle or tissue patches. CPPNs also produce complex geometric patterns, as the activation functions at each node can take on a variety of functions (here: sigmoid, sine, absolute value, negative absolute value, square, negative square, square root, and negative square root). These functions tend to produce regular patterns and features across the coordinate inputs (for example: an absolute value node with an $x$ input would produce left-right symmetry, or a sine node with a $y$ input would produce front-to-back repetition).

This network is optimized to produce high performing morphologies by iterating through various proposed perturbations to it. These include the addition or removal of a node, or edge to the network, as well as the mutation of the weight of any edge or the activation function at each node.

III-F Soft Robot Resolution

As the CPPN genetic encoding is a continuous function (mapping the location of a cell to its cell fate) it may be discretized into a phenotype at any resolution (i.e. creating any number of voxels in the morphology, and a unique controller for each voxel), and in practice this resolution is only limited by computational resources (as more elements are more computationally expensive to simulate). In the default lower-resolution treatment, this discretization occurs over a $5\times 5\times 5$ space. The higher-resolution robots use phenotypes created at a $10\times 10\times 10$ resolution. Note that the distance values are noted in absolute number of voxels, so higher fitness values tend to be produced by phenotypes of higher resolution.

III-G Controllers and their Genetic Encoding

A unique controller is optimized for each muscle cell in the robot’s morphology. For each voxel’s controller, two parameters are optimized (the outputs of a separate CPPN with the same inputs as before). One is the phase offset between each individual cell’s muscle oscillations and that of the global sinusoidal oscillator (which acts as a central pattern generator), and the second is the frequency of this global clock (since CPPNs don’t currently enable global parameters, this is done by averaging the local values at each cell to produce a single global value).

All controllers output a value between $-1$ and $1$ at each time step, which corresponds to a linear change in each dimension of a muscle cell ($\pm 14\%$ of its original length, or $\pm 48\%$ of its original volume), defining a robot’s behavior. Passive tissue cells remain at their original size (though they also deform based on their intrinsic compliance).

While this encoding is simple and straightforward, it has the ability to produce complex behaviors, such as multiple patches of muscle groups that are in sync, counter-sync, or any real valued phase offset from each other. It also has the ability to produce gradually varying sweeps of phase offset, resulting in propagating waves of excitation across large muscle groups. Furthermore, the optimization of the global frequency is able to produce oscillation speeds which are fine tuned to the properties of individual morphologies (such as optimizing to maximize the resonance of soft tissues in appendages).

The encoding of the morphology and the controller of the robot into two separate CPPN networks emphasizes the false dichotomy of robot brains and body plans. However, this explicit separation allows us to make changes specific and isolated to either the brain or the body. This is necessary for the proposed algorithm, as controller readaptation requires iterating through controller changes without affecting the morphology.

III-H Physics Simulation for Evaluation

Once the morphology and controller for a given robot are specified, the fitness (locomotion distance) of that robot is determined by constructing and simulating that robot in the VoxCad soft-body physics simulator [19]. Simulations last for 20 actuation cycles (which may be a variable amount of time, depending on the length of the globally optimized frequency – though this method of normalizing for the number of “steps” taken leads to a more fair comparison than normalizing by the amount of time per simulation).

III-I Morphological Innovation Protection

In our newly proposed method we set the “morphological age” to zero for each new “child” that was the result of a morphological mutation to a current individual in the population. This means that for an individual to have a large value in their age objective, that individual must have been the result of a large number of successive controller mutations without any change to that robot’s body plan. This setup thus allows a simple comparison method for individuals who have had similar amounts of controller (re)adaptation to their current morphologies – as a dominated individual would have to have been out-competed in fitness ability by a morphology that is paired with a controller that is less-well adapted to it. This diversity maintenance mechanism encourages the exploration of new peaks in the rugged landscape of brain-body plans – with the implicit assumption that unique morphologies correspond to peaks in this landscape.

The method of age resets corresponding to morphological mutations to existing individuals differs from the prior technique (e.g. [20, 33]) of inserting completely random individuals, as it allows the improvement in the fitness levels of age-zero individuals over time. In the case of traditional age-fitness optimization, age-zero individuals are drawn from the same distribution of fitness values for random genotypes regardless of when they are created. However, in the case of morphological innovation protection, age-zero individuals are not random and inherit many of the properties of their parents – meaning that they show higher fitness values over time. To demonstrate this empirically, we performed linear regression on the age-zero individuals from Fig. 2 (containing morphological innovation protection), which showed a significant ($p<0.001$) increase in fitness over time (from 18.715 at generation 0 to 22.916 at generation 5000). This confirms a major difference between our proposed technique and the standard approach of age-based diversity maintenance through the introduction of random individuals.

III-J Statistical Analysis

All treatments were performed for 30 independent trials, with random seeds consistent between treatments. All plots show mean values averaged across the most fit individual of 30 trials for each condition with shaded areas representing 95% bootstrapped confidence intervals of this average, and all p-values are generated by a Wilcoxon Rank-Sum Test [37].

IV Results

IV-A The Effect of Morphological Innovation Protection on Fitness

For the task of locomotion ability (a standard task in evolutionary robotics [5]), we first optimize the robots using the traditional method of “greedy” fitness evaluations for our selection criteria (where immediate locomotion ability determines survival in the population of candidate morphologies – i.e. with no “innovation protection”). In this setup, the traditional method produces robots with an average fitness of 21.717 voxels (with 95% bootstrapped confidence (CI) interval of 19.457 to 22.426 voxels).

Additionally, we optimized robots in the same task and environment setup, but this time using “morphological innovation protection” for our selection method – in which individuals can only be out-competed by those with equal or lesser amounts of controller (re)adaptation to their current morphologies. This treatment produces significantly more effective robots ($p=6.067*10^{-6}$), with a mean distance travelled of 31.953 voxels (and 95% CI of 28.157 to 36.511 voxels).

The increase in fitness shows that morphological innovation protection is a more effective way of optimizing robots, yet it does not conclusively demonstrate that the intuition of [8] is correct, as that work demonstrated the asymmetric difficulty in optimizing the morphology of a robot (as compared to optimizing its controller) and drew the hypotheses that this because the morphology encapsulated the controller (acting as a translator between the “language” of the “cognitive” functions and the outside environment). While the above experiment does help to support the intuition that the controller must readapt to a new morphological encoding, it also introduces confounding effects, such as the added population diversity afforded by “protection” and the added dimensionality of the search space from its this protection age – moving search from a single-objective to multi-objective optimization problem.

To tease apart the influence of these two confounds, we present a treatment where the controllers of the robot undergo an equivalent protection to which the morphologies did in the above experiment. In this treatment, individuals can only be out-competed by others whose morphology has had equal or lesser amounts of readaptation to their newly mutated controllers – deemed “controller innovation protection”. This condition provides the potential advantages of multi-dimensional search and the added diversity from temporary reductions in selection pressure. Yet it does not rely on the idea of a broken “morphological language.” In this treatment, robots locomote 22.049 voxels on average (95% CI of 20.726 to 22.641 voxels), which fails to show a significant improvement over the single-objective case of no protection ($p=0.240$), and performs significantly worse ($p=1.211*10^{-4}$) than the protection of morphological innovations.

The fitness trajectories over evolutionary time (Fig. 1) demonstrate a typical logarithmic fitness improvement over the first 1000 generations or so, but then show a stagnation for the traditional optimization procedure without innovation protection. The mean fitness value of the treatment without protection shows no significant improvement ($p=0.085$) from generation 1000 to 5000 (with average fitness values of 20.988 and 21.717 voxels, respectively). Contrary to this, the treatment with morphological innovation protection shows significant improvements over this time ($p=0.013$) from 25.925 at generation 1000 to 31.953 voxels at generation 5000.

The rapid improvement in the controller innovation protection and no protection cases during the first 1000 generations does not contradict the hypotheses of a fragile “morphological language,” as the coupling between the morphology and controller takes time to become established – and would not introduce fragility into the system before then.

IV-B The Effect of Morphological Innovation Protection on Population Stagnation

Perhaps more telling than the average locomotion ability at the end of optimization time is the examination of the optimization process within each individual run. Figs. 2 and 3 represent typical runs, and help to give an intuition of the optimization process. In these figures, each randomly colored line represents a unique morphology, plotted by its locomotion ability over optimization time.

Note the continued improvement in performance of the most fit individual over optimization time in the case of morphological innovation protection (Fig. 2) – which is consistent with the trend seen on average in Fig. 1. This is not seen in the case without innovation protection (Fig. 3), where the best individual was found before generation 2000 – and by generation 1000, fitness has reached 99.6% of its final value.

Consistent with the above observation, we see that the most fit individual in Fig 2 changes rapidly, continually turning over in the trial with morphological innovation protection. As each color in the figure represents a unique morphology, we notice that a wide variety of different morphologies hold the title of “best-so-far”. On average, in runs with morphological innovation protection, 24.179 unique morphologies are the best-so-far at some point in optimization, where the runs without protection show significantly ($p=1.555*10^{-6}$) less turnover of morphologies, with just 10.115 unique robot body plans doing so.

The question of how temporary reduced selection pressure (via the morphological-age dimension) of morphological innovation protection may help to improve overall fitness and continued optimization may be best demonstrated in the pop out box for Fig. 2. Here we see the current best morphology, in teal. This morphology was unable to improve on itself for some time, as we see its fitness value (y-xais) flatlining. This “parent” morphology has a “child,” a new proposed variation of its morphology, highlighted in red. As the original fitness value of this morphology falls below its parent, this individual empirically shows worse performance than its parent – and thus would not be considered as a viable solution in a traditional evolutionary method. However, since this new morphology does not have a controller that is well adapted to it (as the controller is specialized for the previous morphology, in teal), morphological innovation protection does not expect this new robot to immediately outperform its parent, and keeps this individual in consideration as one which could hold long term potential but does not show immediate promise.

Indeed, we see that after a number of controller optimization iterations later (occurring in equal amounts to both the parent and child during this intermediate period), the child morphology (in red) overtakes the parent morphology (in teal) – achieving higher fitness and demonstrating that it did indeed hold a better long term potential than its parent, despite the immediate drop in fitness. As the fitness of the parent is outperformed by the child (which has had less time to fine-tune its controller to its morphology), we assume that the parent is unlikely to be the most promising robot body plan in the long run, and thus remove it from the population.

We see this trend of “overtakes” – where children start out with worse fitness than their parents, but eventually outperform them – continuing throughout this run (as the blue child then overtakes the red parent, and green overtakes blue in the pop out of Fig. 2). We see morphological overtakes significantly ($p\leq 6.939*10^{-10}$) more often in runs with morphological innovation protection (an average of 76.714 overtakes in the first 5000 generations) than without any innovation protection (where there are only 1.432 overtakes) or in the trials with controller innovation protection (where there are just 1.333 morphological overtakes on average).

Interestingly, the number of morphological overtakes in the morphological innovation protection treatment is not significantly different ($p=0.533$) than the number of controller overtakes in the controller innovation protection treatment (where a “child” is a robot with a new controller variation, that readapts its morphology to catch back up to its “parent” controller – which happens 74.542 times on average). Combined with the finding in Fig. 1 that morphological innovation protection outperforms the other two treatments, this suggests a greater potential for the relative importance of morphological overtakes over controller overtakes – and again reinforces the asymmetry between morphologies and controllers from an optimization perspective.

IV-C The Effects of Morphological Innovation Protection on the Progression of Morphologies over Evolutionary Time

Visual inspection of actual morphologies over evolutionary time further supports the proposed method’s improved optimization efficiency and ability to escape local optima. Fig. 4 shows the current best individual at various points over evolutionary time for each of the top 10 runs in the treatment with morphological innovation protection. Note how the fitness values of the robots increase over time (from left to right, and indicated by the color of each robot). Also note how the final morphology of some robots (e.g. runs 24, 27, 18, 11, and 16) result in identical morphologies, despite starting from a range of starting morphologies and not finding this convergent morphology until hundreds or thousands of generations into the optimization process.

In contrast to the sustained turnover of morphologies shown above, Fig. 5 shows the snapshots of the 10 best runs in the treatment without innovation protection. Notice how the colors of the robots tend to show little change over the evolutionary process, mirroring the stagnation shown in Fig. 1. While convergence of the final morphologies is present here as well, the gross morphologies found here (variants of the a full cube with no appendages) are found early on in the search – and often provided in the set of random initial morphologies. In this treatment, gross morphological changes tend to be absent after generation 50 (just 1% into the full 5000 generations).

The differences between Figs. 4 and 5 suggests that the traditional method without morphological innovation protection tends to converge prematurely to morphologies early in the evolutionary search, while morphological innovation protection may allow search to escape these local optima and converge to “more global” optima. The convergence of morphologies across varied initial conditions is even further pronounced in Fig. 6.

IV-D The Potential for Convergence Across Initial Conditions

To explore the question of scale, we apply morphological innovation protection to the evolution of robots with higher resolution morphologies – up to $10^{3}=1000$ voxels rather than the lower resolution morphologies ($5^{3}=125$ voxels) employed in the previous experiments.

The increased number of voxel cells that make up each robot allows for greater expressiveness and finer details in its morphology. However this also presents a challenge for the above algorithm. As the number of cells increases, the effect of changing a single voxel (the minimum morphological variation) is reduced. In the extreme, the concept of readapting controllers since the last “morphological change” is less straightforward – as increasingly small changes can modify minor details of the morphology without affecting its overall function.

To help address the problem of non-functional morphology changes, we introduce a parameter to represent the minimum percentage of voxels that must be varied in order to qualify as a “gross morphological change”. It is important to note that this parameter is specific to the voxel-based soft robot implementation employed in this work – and thus the optimal setting of this parameter is not of great importance for its generalization outside of this soft robot encoding. But the general concept of a threshold for the minimum morphological change is a universal concept that could be applied to any robot instantiation, as necessary.

In the case of robots with higher-resolution morphologies, we find that resetting the “morphological age” of an individual only after a mutation that changes more than 20% of their voxels produces optimal results. The value was found via a parameter sweep (of minimum age-resets of 0%, 5%, 10%, 20%, 30%, 40%, and 50% of voxels changed). We also investigated a minimum change threshold for controller innovation protection and found that its benefit falls as the threshold increases (showing optimal performance with no threshold), so we ignore the threshold for minimum controller change here.

In this setting, basic morphological innovation protection (with no threshold) travels significantly farther (60.728 voxels) than both the case with no protection (32.575 voxels, $p=1.208*10^{-4}$) and the case with controller innovation protection (35.375 voxels, $p=0.018$). While trials employing the threshold on morphological changes did achieve higher average fitness than the case of morphological innovation protection without a threshold (where robots traveled 43.340 voxels on average), this difference was not significant at the $p=0.05$ level ($p=0.063$).

As a side note, in other more difficult trials (where the robots need to optimize a closed-loop neural network controller – not depicted visually due to space constraints), trials with the morphological change threshold significantly outperform all other treatments (as they travel 31.313 voxels – while trials with no protection, controller protection, and morphological innovation protection with no threshold travel 13.941, 15.356, and 17.224 voxels, respectively; all $p\leq 7.287*10^{-8}$).

Visualization of these soft robot morphologies over time in the trials with a 20% morphological change threshold is depicted in Fig. 6 and compared to the case with no protection in Fig. 7. The ability of evolution to converge to the same high-performing morphology across many independent trials, despite starting from different initial conditions suggests that (in this particular soft robot implementation) the inclusion of thresholded morphological innovation protection is able to escape the local optima around these starting conditions and find “more global” optima in this search space.

In the case without any protection, search stagnates quickly and again appears unable to escape the local optima near its initial conditions (Fig. 7).

Interestingly, the low resolution soft robot implementation (in Figs. 4 and 5) does not benefit from the inclusion of a threshold. As presumably the more discrete construction of these robots mean that all mutations are large enough to create a meaningful “morphological change”.

V Discussion

The above results demonstrate a new method for entire robot evolution (i.e. both brain and body-plan) that is more scalable in terms of continued optimization for longer periods of time and better resulting fitness than the traditional evolutionary methods. This method for “morphological innovation protection” helps prevent premature convergence to the many local optima which appear to be present in the rugged search space of robot morphologies and controllers [8].

The hypothesis from [8] that the fragile co-optimization of brain and body plan caused by specialization of one sub-component to the other is consistent with the findings above. This work also reveals that there is an asymmetry between the brain and body plan: protecting innovations to the morphology leads to more effective optimization that protecting innovations to the controller.

The benefits of the temporarily reduced selection pressure provided by morphological innovation protection suggests that the long-term potential and immediate fitness impact of a morphological mutation are not always correlated. Thus, we require a form of diversity maintenance to help evolution to rate proposed solutions based on their long-term potential rather than on immediate fitness impact. As was shown here, using morphological innovation protection for this purpose can help to reduce premature convergence in the search space and stagnation at suboptimal values.

We believe this to be the first example of a design automation algorithm for robotics that considers the interdependence of neural controllers and body plans arising from the psychological theory of embodied cognition [30] and uses this intuition to propose a method to escape local optima in the fitness landscape.

Despite the significant improvement to our ability to simultaneously optimize the brain and body plan of embodied robots, there is much work still to be done. Firstly, the proposed method was only applied to one class of robot. This class may actually represent the simplest form of brain-body co-optimization because the distributed sensing, actuation, and information processing that the cellular soft robot paradigm was designed to possess helps to blur the line between physical interactions of the morphology with the environment and information processing of a controller [10, 9].

In the case of centralized controllers and robots composed of rigid components, topological (rather than parametric) changes to the cognitive architecture would be required for control readaptation if morphological mutations add or remove physical components (e.g. limbs). Future work should explore the effect of morphological innovation protection in such a paradigm, where there is the potential for morphological changes to more drastically change the function of the robot – and thus for readaptation to those morphologies to play an even more critical role in optimization.

In these experiments, the genotype encoding of the soft robots was modularized such that one part of the genome dictated the shape and material properties of the robot and a separate part encoded the actuations in the form of volumetric deformations of the voxels during behavior. In these robots (and those with even more complex phenotypes), one can envision different splits as to what constitutes morphology and control – and what qualifies as a “morphological change” in this proposed algorithm. It would be of interest to investigate the effect of various splitting points of this dichotomy on the efficacy of morphological innovation protection.

Furthermore, the very idea of a distinct split between body and brain is a false dichotomy, as information processing and physical processes happen throughout the agent [2]. Rather than various mutation operators which affect only the “brain” or the “body-plan” of a robot, we desire agents that grown and develop as the result of rich crosstalk and feedback loops between the processes responsible for creating both of these components (creating complex adaptive systems like we see in biological organisms).

How this proposed method relates to, and informs, biological analogs not entirely clear, but some rough analogies can be drawn. For example, the observation that brains learn and adapt at a much faster rate than bodies grow fits into this paradigm, as does the even slower change of gross morphological features over evolutionary time. In this way, the readaptation of controller strategies for varying body plans is built into the neuroplasticity of the brain.

One could also imagine periods of evolutionary time when an entire species is under relatively little morphological selection pressure before an environmental shock suddenly reapplies that pressure. Also possible are periods of a single individual’s lifetime when selection pressure varies: For example, human infants may not be selected highly for their locomotion speed, as their parents tend to physically carry and protect them early in life.

We do not believe that the methods introduced here are restricted to this particular domain. The above algorithm is simple to implement (requiring only: an age counter, a check for variations in brain and/or body for each mutation, and –optionally – a criterion for the minimal gross morphological change), and thus we believe it will be widely applicable. Future work will test this supposition.

Due to the recent interest in co-optimization of neural network topology and weights [15, 28], we should also note that this work – an agent’s controller embodied within its morphology – is closely related to that of neural network’s weights embodied within its topology. Future work will show whether the method proposed here will show similar optimization gains in the design of neural network topologies as well.

VI Conclusion

We demonstrate an example of a robot design automation algorithm that considers the interdependence of neural controllers and body plans (due to the theory of embodied cognition) on the optimization process. We use this intuition to temporarily reduce selection pressure on newly mutated robot morphologies, thus allowing the agents to readapt their controllers and better escape local optima in the fitness landscape. We have shown that this technique – deemed “morphological innovation protection” – produces evolutionary optimization which delays premature convergence and stagnation, and results in more efficient evolved robots. We showcase the ability of this technique to escape local optima in the search space by demonstrating the convergence to a similar morphology across many independent trials from randomly initial conditions. While we hope that this technique will be surpassed in the future by a developmental process with feedback loops between the body and brain, we propose the above algorithm as a short term improvement over the current techniques for the co-optimization of morphology and control in virtual creatures.

VII Acknowledgment

We thank NASA Space Technology Research Fellowship #NNX13AL37H and Army Research Office Contract #W911NF1610304 for support, Steve Strogatz for advice, Kathryn Miller for copy editing, and the UVM Morphology, Evolution & Cognition Lab for feedback.