Cracking the neural code for word recognition in convolutional neural networks

TL;DR

This study investigates how deep neural networks recognize written words invariantly, revealing specialized units that encode letter identity and position, similar to the human brain's visual word form area.

Contribution

The paper demonstrates that trained neural networks develop units that encode letter identities and positions, providing mechanistic insights into invariant word recognition.

Findings

Units become specialized for word recognition after training

Units encode specific letter identities and positional information

Pooling across layers supports invariant recognition

Abstract

Learning to read places a strong challenge on the visual system. Years of expertise lead to a remarkable capacity to separate highly similar letters and encode their relative positions, thus distinguishing words such as FORM and FROM, invariantly over a large range of sizes and absolute positions. How neural circuits achieve invariant word recognition remains unknown. Here, we address this issue by training deep neural network models to recognize written words and then analyzing how reading-specialized units emerge and operate across different layers of the network. With literacy, a small subset of units becomes specialized for word recognition in the learned script, similar to the "visual word form area" of the human brain. We show that these units are sensitive to specific letter identities and their distance from the blank space at the left or right of a word, thus acting as "space bigrams". These units specifically encode ordinal positions and operate by pooling across low and high-frequency detector units from early layers of the network. The proposed neural code provides a mechanistic insight into how information on letter identity and position is extracted and allow for invariant word recognition, and leads to predictions for reading behavior, error patterns, and the neurophysiology of reading.