Adapting Brain-Like Neural Networks for Modeling Cortical Visual Prostheses

TL;DR

This paper explores using brain-like convolutional neural networks to predict visual percepts from cortical prostheses, aiming to improve artificial vision and understand neural coding of vision.

Contribution

It introduces a proof-of-concept model that decodes CNN activations to generate realistic visual percepts from electrical stimulation in the visual cortex.

Findings

Qualitatively accurate phosphenes produced by the model

Model's predictions are comparable to patient-reported percepts

First step towards brain-like models for cortical visual prostheses

Abstract

Cortical prostheses are devices implanted in the visual cortex that attempt to restore lost vision by electrically stimulating neurons. Currently, the vision provided by these devices is limited, and accurately predicting the visual percepts resulting from stimulation is an open challenge. We propose to address this challenge by utilizing 'brain-like' convolutional neural networks (CNNs), which have emerged as promising models of the visual system. To investigate the feasibility of adapting brain-like CNNs for modeling visual prostheses, we developed a proof-of-concept model to predict the perceptions resulting from electrical stimulation. We show that a neurologically-inspired decoding of CNN activations produces qualitatively accurate phosphenes, comparable to phosphenes reported by real patients. Overall, this is an essential first step towards building brain-like models of electrical stimulation, which may not just improve the quality of vision provided by cortical prostheses but could also further our understanding of the neural code of vision.