A Deep Learning-based in silico Framework for Optimization on Retinal Prosthetic Stimulation

TL;DR

This paper introduces a neural network framework that optimizes retinal prosthetic stimulation perceptions in silico, improving image perception quality through an end-to-end trainable pipeline.

Contribution

It presents a novel deep learning-based in silico framework combining trainable encoders and pre-trained models for retinal prosthetic optimization.

Findings

Significant improvement in perception quality with a 36.17% boost in weighted F1-Score.

The neural network encoder outperforms trivial downsampling methods.

End-to-end training allows fine-tuning of perception quality.

Abstract

We propose a neural network-based framework to optimize the perceptions simulated by the in silico retinal implant model pulse2percept. The overall pipeline consists of a trainable encoder, a pre-trained retinal implant model and a pre-trained evaluator. The encoder is a U-Net, which takes the original image and outputs the stimulus. The pre-trained retinal implant model is also a U-Net, which is trained to mimic the biomimetic perceptual model implemented in pulse2percept. The evaluator is a shallow VGG classifier, which is trained with original images. Based on 10,000 test images from the MNIST dataset, we show that the convolutional neural network-based encoder performs significantly better than the trivial downsampling approach, yielding a boost in the weighted F1-Score by 36.17% in the pre-trained classifier with 6x10 electrodes. With this fully neural network-based encoder, the quality of the downstream perceptions can be fine-tuned using gradient descent in an end-to-end fashion.