GNSP: Gradient Null Space Projection for Preserving Cross-Modal Alignment in VLMs Continual Learning

TL;DR

This paper introduces GNSP, a continual learning method for vision-language models that prevents forgetting by projecting gradients onto a null space, preserving cross-modal alignment and zero-shot capabilities.

Contribution

GNSP is a novel gradient projection technique that maintains cross-modal alignment in VLMs during continual learning without rehearsal or architecture changes.

Findings

Achieved state-of-the-art performance on the MTIL benchmark.

Successfully preserved the original modality gap and cross-modal retrieval performance.

Effectively maintained the structure of the multimodal embedding space.

Abstract

Contrastive Language-Image Pretraining has demonstrated remarkable zero-shot generalization by aligning visual and textual modalities in a shared embedding space. However, when continuously fine-tuned on diverse tasks, CLIP suffers from catastrophic forgetting and degradation of its embedding alignment, undermining its zero-shot capabilities. In this work, we propose Gradient Null Space Projection (GNSP), an efficient continual learning method that projects task-specific gradients onto the null space of previously learned knowledge. This orthogonal projection mathematically prevents interference with previous tasks without relying on rehearsal or architectural modification. Furthermore, to preserve the inherent generalization property of CLIP, we introduce knowledge distillation and combine it with a modality alignment preservation loss inspired by CLIP pre-training to stabilize the structure of the multimodal embedding space during fine-tuning. On the MTIL benchmark consisting of 11 tasks, our method achieved SOTA performance on both the Average and Last key metrics. More importantly, experiments show that our method successfully maintains the original modality gap and cross-modal retrieval performance of CLIP, confirming its effectiveness in maintaining a robust visual-language space throughout the continual learning process.