Seeing the imagined: a latent functional alignment in visual imagery decoding from fMRI data

TL;DR

This paper introduces a latent functional alignment method to improve visual imagery decoding from fMRI data by leveraging perception-trained models and semantic retrieval augmentation, demonstrating consistent improvements across subjects.

Contribution

It proposes a novel latent alignment approach that enhances imagery decoding by mapping activity into a pretrained model's space and uses retrieval-based augmentation for better performance.

Findings

Latent functional alignment improves semantic reconstruction metrics.

The method enables above-chance decoding from multiple cortical regions.

It outperforms baseline methods in perception-to-imagery transfer tasks.

Abstract

Recent progress in visual brain decoding from fMRI has been enabled by large-scale datasets such as the Natural Scenes Dataset (NSD) and powerful diffusion-based generative models. While current pipelines are primarily optimized for perception, their performance under mental-imagery remains less well understood. In this work, we study how a state-of-the-art (SOTA) perception decoder (DynaDiff) can be adapted to reconstruct imagined content from the Imagery-NSD benchmark. We propose a latent functional alignment approach that maps imagery-evoked activity into the pretrained model's conditioning space, while keeping the remaining components frozen. To mitigate the limited amount of matched imagery-perception supervision, we further introduce a retrieval-based augmentation strategy that selects semantically related NSD perception trials. Across four subjects, latent functional alignment consistently improves high-level semantic reconstruction metrics relative to the frozen pretrained baseline and a voxel-space ridge alignment baseline, and enables above-chance decoding from multiple cortical regions. These results suggest that semantic structure learned from perception can be leveraged to stabilize and improve visual imagery decoding under out-of-distribution conditions.