Exploration and Comparison of Deep Learning Architectures to Predict Brain Response to Realistic Pictures

TL;DR

This study explores various deep learning architectures, especially using CLIP embeddings, to predict brain responses to realistic images, highlighting the challenges and effective strategies in modeling high-dimensional neural data.

Contribution

The paper systematically compares different deep learning models for brain response prediction, demonstrating that simple ROI-specific linear models with CLIP embeddings perform best.

Findings

Simple ROI-specific models outperform complex architectures.

Using CLIP embeddings improves prediction accuracy.

Challenges include high dimensionality and noisy data.

Abstract

We present an exploration of machine learning architectures for predicting brain responses to realistic images on occasion of the Algonauts Challenge 2023. Our research involved extensive experimentation with various pretrained models. Initially, we employed simpler models to predict brain activity but gradually introduced more complex architectures utilizing available data and embeddings generated by large-scale pre-trained models. We encountered typical difficulties related to machine learning problems, e.g. regularization and overfitting, as well as issues specific to the challenge, such as difficulty in combining multiple input encodings, as well as the high dimensionality, unclear structure, and noisy nature of the output. To overcome these issues we tested single edge 3D position-based, multi-region of interest (ROI) and hemisphere predictor models, but we found that employing multiple simple models, each dedicated to a ROI in each hemisphere of the brain of each subject, yielded the best results - a single fully connected linear layer with image embeddings generated by CLIP as input. While we surpassed the challenge baseline, our results fell short of establishing a robust association with the data.