Neural encoding with affine feature response transforms

TL;DR

This paper introduces AFRT, a novel neural encoding method inspired by neuroscience, which improves efficiency and interpretability by modeling receptive fields with affine transforms and localized responses, reducing parameters significantly.

Contribution

The paper presents AFRT, a new neural encoding approach that incorporates retinotopic organization, enabling more efficient and interpretable models with fewer parameters.

Findings

AFRT reduces redundant computations in neural encoding models.

AFRT achieves comparable or better performance with fewer parameters.

Receptive fields modeled by AFRT align well with biological data.

Abstract

Current linearizing encoding models that predict neural responses to sensory input typically neglect neuroscience-inspired constraints that could enhance model efficiency and interpretability. To address this, we propose a new method called affine feature response transform (AFRT), which exploits the brain's retinotopic organization. Applying AFRT to encode multi-unit activity in areas V1, V4, and IT of the macaque brain, we demonstrate that AFRT reduces redundant computations and enhances the performance of current linearizing encoding models by segmenting each neuron's receptive field into an affine retinal transform, followed by a localized feature response. Remarkably, by factorizing receptive fields into a sequential affine component with three interpretable parameters (for shifting and scaling) and response components with a small number of feature weights per response, AFRT achieves encoding with orders of magnitude fewer parameters compared to unstructured models. We show that the retinal transform of each neuron's encoding agrees well with the brain's receptive field. Together, these findings suggest that this new subset within spatial transformer network can be instrumental in neural encoding models of naturalistic stimuli.