A Frequency-Domain Encoding for Neuroevolution

TL;DR

This paper introduces a frequency-domain encoding for neuroevolution that uses Fourier coefficients to represent network weights, significantly reducing search space dimensionality and improving convergence in complex reinforcement learning tasks.

Contribution

It presents a novel frequency-based indirect encoding method for neuroevolution, enabling efficient search in high-dimensional network spaces.

Findings

Frequency-domain encoding reduces search space by up to 100 times.

Encoding accelerates convergence in reinforcement learning tasks.

Results show more general solutions with the frequency approach.

Abstract

Neuroevolution has yet to scale up to complex reinforcement learning tasks that require large networks. Networks with many inputs (e.g. raw video) imply a very high dimensional search space if encoded directly. Indirect methods use a more compact genotype representation that is transformed into networks of potentially arbitrary size. In this paper, we present an indirect method where networks are encoded by a set of Fourier coefficients which are transformed into network weight matrices via an inverse Fourier-type transform. Because there often exist network solutions whose weight matrices contain regularity (i.e. adjacent weights are correlated), the number of coefficients required to represent these networks in the frequency domain is much smaller than the number of weights (in the same way that natural images can be compressed by ignore high-frequency components). This "compressed" encoding is compared to the direct approach where search is conducted in the weight space on the high-dimensional octopus arm task. The results show that representing networks in the frequency domain can reduce the search-space dimensionality by as much as two orders of magnitude, both accelerating convergence and yielding more general solutions.