COBRA: A Continual Learning Approach to Vision-Brain Understanding

TL;DR

COBRA is a novel continual learning framework for vision-brain understanding that effectively mitigates catastrophic forgetting by combining shared and subject-specific modules, leading to state-of-the-art results in fMRI-based vision reconstruction.

Contribution

The paper introduces COBRA, a new continual learning framework with three modules that preserve shared knowledge and learn subject-specific patterns in fMRI data.

Findings

COBRA outperforms previous methods in continual learning tasks.

It effectively reduces catastrophic forgetting in vision-brain understanding.

Achieves state-of-the-art performance in vision-brain reconstruction.

Abstract

Vision-Brain Understanding (VBU) aims to extract visual information perceived by humans from brain activity recorded through functional Magnetic Resonance Imaging (fMRI). Despite notable advancements in recent years, existing studies in VBU continue to face the challenge of catastrophic forgetting, where models lose knowledge from prior subjects as they adapt to new ones. Addressing continual learning in this field is, therefore, essential. This paper introduces a novel framework called Continual Learning for Vision-Brain (COBRA) to address continual learning in VBU. Our approach includes three novel modules: a Subject Commonality (SC) module, a Prompt-based Subject Specific (PSS) module, and a transformer-based module for fMRI, denoted as MRIFormer module. The SC module captures shared vision-brain patterns across subjects, preserving this knowledge as the model encounters new subjects, thereby reducing the impact of catastrophic forgetting. On the other hand, the PSS module learns unique vision-brain patterns specific to each subject. Finally, the MRIFormer module contains a transformer encoder and decoder that learns the fMRI features for VBU from common and specific patterns. In a continual learning setup, COBRA is trained in new PSS and MRIFormer modules for new subjects, leaving the modules of previous subjects unaffected. As a result, COBRA effectively addresses catastrophic forgetting and achieves state-of-the-art performance in both continual learning and vision-brain reconstruction tasks, surpassing previous methods.