Learning Brain Representation with Hierarchical Visual Embeddings

TL;DR

This paper introduces a novel method for decoding visual information from brain signals by aligning hierarchical visual embeddings with brain activity, utilizing multiple pre-trained encoders and a fusion prior to improve accuracy and fidelity.

Contribution

It proposes a multi-scale, hierarchical brain-image alignment strategy with a fusion prior, advancing the understanding of visual representations in the human brain.

Findings

Improved retrieval accuracy of visual stimuli from brain signals.

Enhanced reconstruction fidelity of visual images from neural data.

Effective alignment of multi-scale visual features with brain activity.

Abstract

Decoding visual representations from brain signals has attracted significant attention in both neuroscience and artificial intelligence. However, the degree to which brain signals truly encode visual information remains unclear. Current visual decoding approaches explore various brain-image alignment strategies, yet most emphasize high-level semantic features while neglecting pixel-level details, thereby limiting our understanding of the human visual system. In this paper, we propose a brain-image alignment strategy that leverages multiple pre-trained visual encoders with distinct inductive biases to capture hierarchical and multi-scale visual representations, while employing a contrastive learning objective to achieve effective alignment between brain signals and visual embeddings. Furthermore, we introduce a Fusion Prior, which learns a stable mapping on large-scale visual data and subsequently matches brain features to this pre-trained prior, thereby enhancing distributional consistency across modalities. Extensive quantitative and qualitative experiments demonstrate that our method achieves a favorable balance between retrieval accuracy and reconstruction fidelity.