Decoding the Echoes of Vision from fMRI: Memory Disentangling for Past Semantic Information

TL;DR

This paper introduces a novel contrastive learning approach to decode and disentangle past visual memories from fMRI signals, enhancing understanding of memory encoding during continuous visual processing.

Contribution

It proposes a new Memory Disentangling task and a contrastive learning method to separate current and past information in fMRI data, addressing interference issues.

Findings

Effective disentanglement of past and current memory signals in fMRI data

Improved decoding accuracy of visual memories from brain signals

Potential applications in brain-computer interfaces

Abstract

The human visual system is capable of processing continuous streams of visual information, but how the brain encodes and retrieves recent visual memories during continuous visual processing remains unexplored. This study investigates the capacity of working memory to retain past information under continuous visual stimuli. And then we propose a new task Memory Disentangling, which aims to extract and decode past information from fMRI signals. To address the issue of interference from past memory information, we design a disentangled contrastive learning method inspired by the phenomenon of proactive interference. This method separates the information between adjacent fMRI signals into current and past components and decodes them into image descriptions. Experimental results demonstrate that this method effectively disentangles the information within fMRI signals. This research could advance brain-computer interfaces and mitigate the problem of low temporal resolution in fMRI.