Coupling Visual Semantics of Artificial Neural Networks and Human Brain Function via Synchronized Activations

TL;DR

This paper introduces Sync-ACT, a novel framework that links artificial neural network representations with human brain semantics using nfMRI data, revealing significant correlations and potential for brain-inspired AI.

Contribution

The study presents the first method to semantically annotate ANN neurons with human brain data, establishing a new link between artificial and biological neural representations.

Findings

Significant correlation between CNN visual semantics and human brain networks.

CNN's similarity to brain representations correlates with classification performance.

Sync-ACT effectively couples ANN and BNN semantic spaces.

Abstract

Artificial neural networks (ANNs), originally inspired by biological neural networks (BNNs), have achieved remarkable successes in many tasks such as visual representation learning. However, whether there exists semantic correlations/connections between the visual representations in ANNs and those in BNNs remains largely unexplored due to both the lack of an effective tool to link and couple two different domains, and the lack of a general and effective framework of representing the visual semantics in BNNs such as human functional brain networks (FBNs). To answer this question, we propose a novel computational framework, Synchronized Activations (Sync-ACT), to couple the visual representation spaces and semantics between ANNs and BNNs in human brain based on naturalistic functional magnetic resonance imaging (nfMRI) data. With this approach, we are able to semantically annotate the neurons in ANNs with biologically meaningful description derived from human brain imaging for the first time. We evaluated the Sync-ACT framework on two publicly available movie-watching nfMRI datasets. The experiments demonstrate a) the significant correlation and similarity of the semantics between the visual representations in FBNs and those in a variety of convolutional neural networks (CNNs) models; b) the close relationship between CNN's visual representation similarity to BNNs and its performance in image classification tasks. Overall, our study introduces a general and effective paradigm to couple the ANNs and BNNs and provides novel insights for future studies such as brain-inspired artificial intelligence.