What are Neural Networks made of?

TL;DR

This paper hypothesizes that neural network training can be viewed as a form of Genetic Programming, offering a new perspective to understand the success and architecture performance of deep learning models.

Contribution

It introduces the novel idea that neural network training resembles Genetic Programming, providing a new framework to analyze neural network behavior and architecture effectiveness.

Findings

Neural network training may be interpreted as Genetic Programming.

This perspective could explain why certain architectures outperform others.

Potential new methods for neural network analysis and design.

Abstract

The success of Deep Learning methods is not well understood, though various attempts at explaining it have been made, typically centered on properties of stochastic gradient descent. Even less clear is why certain neural network architectures perform better than others. We provide a potential opening with the hypothesis that neural network training is a form of Genetic Programming.