ReWaRD: Retinal Waves for Pre-Training Artificial Neural Networks Mimicking Real Prenatal Development

TL;DR

This paper introduces a biologically inspired pre-training method for neural networks using simulated retinal waves, which enhances feature similarity to primate visual cortex and matches state-of-the-art performance without dataset biases.

Contribution

It presents a novel pre-training approach based on simulated retinal waves, mimicking early visual development, and demonstrates its effectiveness in improving neural network features and performance.

Findings

Pre-training with retinal waves produces features similar to primate V1.

Performance gains are comparable to traditional pre-training pipelines.

The method reduces dataset biases inherent in conventional datasets.

Abstract

Computational models trained on a large amount of natural images are the state-of-the-art to study human vision - usually adult vision. Computational models of infant vision and its further development are gaining more and more attention in the community. In this work we aim at the very beginning of our visual experience - pre- and post-natal retinal waves which suggest to be a pre-training mechanism for the primate visual system at a very early stage of development. We see this approach as an instance of biologically plausible data driven inductive bias through pre-training. We built a computational model that mimics this development mechanism by pre-training different artificial convolutional neural networks with simulated retinal wave images. The resulting features of this biologically plausible pre-training closely match the V1 features of the primate visual system. We show that the performance gain by pre-training with retinal waves is similar to a state-of-the art pre-training pipeline. Our framework contains the retinal wave generator, as well as a training strategy, which can be a first step in a curriculum learning based training diet for various models of development. We release code, data and trained networks to build the basis for future work on visual development and based on a curriculum learning approach including prenatal development to support studies of innate vs. learned properties of the primate visual system. An additional benefit of our pre-trained networks for neuroscience or computer vision applications is the absence of biases inherited from datasets like ImageNet.