BIMM: Brain Inspired Masked Modeling for Video Representation Learning

TL;DR

BIMM is a brain-inspired framework that uses dual-branch masked modeling with shared parameters to learn comprehensive video representations, inspired by the ventral and dorsal pathways of the human visual system.

Contribution

This work introduces a novel dual-branch masked modeling approach with partial parameter sharing, inspired by human visual pathways, for improved video representation learning.

Findings

Outperforms state-of-the-art methods on video tasks

Effective in capturing both object and motion information

Demonstrates the benefit of brain-inspired architecture

Abstract

The visual pathway of human brain includes two sub-pathways, ie, the ventral pathway and the dorsal pathway, which focus on object identification and dynamic information modeling, respectively. Both pathways comprise multi-layer structures, with each layer responsible for processing different aspects of visual information. Inspired by visual information processing mechanism of the human brain, we propose the Brain Inspired Masked Modeling (BIMM) framework, aiming to learn comprehensive representations from videos. Specifically, our approach consists of ventral and dorsal branches, which learn image and video representations, respectively. Both branches employ the Vision Transformer (ViT) as their backbone and are trained using masked modeling method. To achieve the goals of different visual cortices in the brain, we segment the encoder of each branch into three intermediate blocks and reconstruct progressive prediction targets with light weight decoders. Furthermore, drawing inspiration from the information-sharing mechanism in the visual pathways, we propose a partial parameter sharing strategy between the branches during training. Extensive experiments demonstrate that BIMM achieves superior performance compared to the state-of-the-art methods.