Spatial encoding of BOLD fMRI time series for categorizing static images across visual datasets: A pilot study on human vision

TL;DR

This study explores how brain activity differs when viewing various images by encoding fMRI data into 2D representations and classifying them with CNNs, revealing insights into visual processing across datasets.

Contribution

It introduces a novel approach of spatially encoding fMRI time series into 2D images using GAF and MTF for improved image categorization across datasets.

Findings

Parallel CNN outperforms other models with 7% higher accuracy.

2D encoding improves classification of brain responses to diverse images.

Baseline established for studying neural differences in visual perception.

Abstract

Functional MRI (fMRI) is widely used to examine brain functionality by detecting alteration in oxygenated blood flow that arises with brain activity. In this study, complexity specific image categorization across different visual datasets is performed using fMRI time series (TS) to understand differences in neuronal activities related to vision. Publicly available BOLD5000 dataset is used for this purpose, containing fMRI scans while viewing 5254 images of diverse categories, drawn from three standard computer vision datasets: COCO, ImageNet and SUN. To understand vision, it is important to study how brain functions while looking at different images. To achieve this, spatial encoding of fMRI BOLD TS has been performed that uses classical Gramian Angular Field (GAF) and Markov Transition Field (MTF) to obtain 2D BOLD TS, representing images of COCO, Imagenet and SUN. For classification, individual GAF and MTF features are fed into regular CNN. Subsequently, parallel CNN model is employed that uses combined 2D features for classifying images across COCO, Imagenet and SUN. The result of 2D CNN models is also compared with 1D LSTM and Bi-LSTM that utilizes raw fMRI BOLD signal for classification. It is seen that parallel CNN model outperforms other network models with an improvement of 7% for multi-class classification. Clinical relevance- The obtained result of this analysis establishes a baseline in studying how differently human brain functions while looking at images of diverse complexities.