Generalizability analysis of deep learning predictions of human brain responses to augmented and semantically novel visual stimuli

TL;DR

This study evaluates the generalization of neural network-based brain encoding models in predicting human visual cortex responses to augmented and novel stimuli, highlighting their potential for optimizing visual augmentations in AR/VR.

Contribution

It demonstrates the ability of top brain encoding models to predict neural responses to unseen and semantically novel visual stimuli, advancing understanding of model generalization in brain response prediction.

Findings

Models can predict responses to unseen stimuli

Brain encoders estimate effects of image augmentations

Evidence supports model generalization capabilities

Abstract

The purpose of this work is to investigate the soundness and utility of a neural network-based approach as a framework for exploring the impact of image enhancement techniques on visual cortex activation. In a preliminary study, we prepare a set of state-of-the-art brain encoding models, selected among the top 10 methods that participated in The Algonauts Project 2023 Challenge [16]. We analyze their ability to make valid predictions about the effects of various image enhancement techniques on neural responses. Given the impossibility of acquiring the actual data due to the high costs associated with brain imaging procedures, our investigation builds up on a series of experiments. Specifically, we analyze the ability of brain encoders to estimate the cerebral reaction to various augmentations by evaluating the response to augmentations targeting objects (i.e., faces and words) with known impact on specific areas. Moreover, we study the predicted activation in response to objects unseen during training, exploring the impact of semantically out-of-distribution stimuli. We provide relevant evidence for the generalization ability of the models forming the proposed framework, which appears to be promising for the identification of the optimal visual augmentation filter for a given task, model-driven design strategies as well as for AR and VR applications.