Bisecle: Binding and Separation in Continual Learning for Video Language Understanding

TL;DR

Bisecle introduces a hippocampus-inspired continual learning method for video-language models, improving memory retention and task adaptation in evolving video understanding tasks with minimal parameter updates.

Contribution

The paper proposes Bisecle, a novel continual learning framework inspired by hippocampal mechanisms, incorporating multi-directional supervision and contrastive prompt learning for efficient video-language model adaptation.

Findings

Bisecle reduces catastrophic forgetting in VideoQA tasks.

It enhances cross-task generalization in continual learning scenarios.

The method demonstrates robustness across multiple benchmarks.

Abstract

Frontier vision-language models (VLMs) have made remarkable improvements in video understanding tasks. However, real-world videos typically exist as continuously evolving data streams (e.g., dynamic scenes captured by wearable glasses), necessitating models to continually adapt to shifting data distributions and novel scenarios. Considering the prohibitive computational costs of fine-tuning models on new tasks, usually, a small subset of parameters is updated while the bulk of the model remains frozen. This poses new challenges to existing continual learning frameworks in the context of large multimodal foundation models, i.e., catastrophic forgetting and update conflict. While the foundation models struggle with parameter-efficient continual learning, the hippocampus in the human brain has evolved highly efficient mechanisms for memory formation and consolidation. Inspired by the rapid Binding and pattern separation mechanisms in the hippocampus, in this work, we propose Bisecle for video-language continual learning, where a multi-directional supervision module is used to capture more cross-modal relationships and a contrastive prompt learning scheme is designed to isolate task-specific knowledge to facilitate efficient memory storage. Binding and separation processes further strengthen the ability of VLMs to retain complex experiences, enabling robust and efficient continual learning in video understanding tasks. We perform a thorough evaluation of the proposed Bisecle, demonstrating its ability to mitigate forgetting and enhance cross-task generalization on several VideoQA benchmarks.