# An Evolutionary Algorithm of Linear complexity: Application to Training of Deep Neural Networks

## TL;DR

This paper introduces a new evolutionary algorithm with linear complexity that efficiently trains deep neural networks with millions of parameters, outperforming traditional methods in high-dimensional settings.

## Contribution

A novel evolutionary algorithm with linear complexity is proposed for training deep neural networks, enabling practical optimization in extremely high-dimensional spaces.

## Key findings

- The new algorithm requires O(n) operations and memory.
- It delivers competitive solutions for training RBMs with over one million variables.
- It outperforms CMA-ES and Contrastive Divergence in high-dimensional training tasks.

## Abstract

The performance of deep neural networks, such as Deep Belief Networks formed by Restricted Boltzmann Machines (RBMs), strongly depends on their training, which is the process of adjusting their parameters. This process can be posed as an optimization problem over n dimensions. However, typical networks contain tens of thousands of parameters, making this a High-Dimensional Problem (HDP). Although different optimization methods have been employed for this goal, the use of most of the Evolutionary Algorithms (EAs) becomes prohibitive due to their inability to deal with HDPs. For instance, the Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES) which is regarded as one of the most effective EAs, exhibits the enormous disadvantage of requiring $O(n^2)$ memory and operations, making it unpractical for problems with more than a few hundred variables. In this paper, we introduce a novel EA that requires $O(n)$ operations and memory, but delivers competitive solutions for the training stage of RBMs with over one million variables, when compared against CMA-ES and the Contrastive Divergence algorithm, which is the standard method for training RBMs.