The Algonauts Project 2023 Challenge: UARK-UAlbany Team Solution

TL;DR

This paper describes a novel ensemble approach using pretrained and fine-tuned brain encoders to predict neural responses to complex natural images, advancing brain modeling techniques.

Contribution

We developed a two-step training ensemble of brain encoders, combining pretrained and subject-specific fine-tuning, for improved prediction of brain responses to visual stimuli.

Findings

Achieved accurate prediction of brain responses across the entire visual cortex.

Demonstrated the effectiveness of ensemble models with diverse training strategies.

Provided open-source code for reproducibility and further research.

Abstract

This work presents our solutions to the Algonauts Project 2023 Challenge. The primary objective of the challenge revolves around employing computational models to anticipate brain responses captured during participants' observation of intricate natural visual scenes. The goal is to predict brain responses across the entire visual brain, as it is the region where the most reliable responses to images have been observed. We constructed an image-based brain encoder through a two-step training process to tackle this challenge. Initially, we created a pretrained encoder using data from all subjects. Next, we proceeded to fine-tune individual subjects. Each step employed different training strategies, such as different loss functions and objectives, to introduce diversity. Ultimately, our solution constitutes an ensemble of multiple unique encoders. The code is available at https://github.com/uark-cviu/Algonauts2023