Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models

TL;DR

This paper presents a novel continual learning framework for vision-language models that uses specialized visual projectors and expert recommendation to better incorporate instructions and adapt to new tasks without forgetting previous knowledge.

Contribution

It introduces a mixture of visual projectors with expert recommendation and pruning strategies to improve instruction grounding and task adaptation in continual learning.

Findings

Outperforms existing continual learning methods on diverse tasks.

Generates instruction-following responses effectively.

Reduces interference from irrelevant experts.

Abstract

Continual learning enables pre-trained generative vision-language models (VLMs) to incorporate knowledge from new tasks without retraining data from previous ones. Recent methods update a visual projector to translate visual information for new tasks, connecting pre-trained vision encoders with large language models. However, such adjustments may cause the models to prioritize visual inputs over language instructions, particularly learning tasks with repetitive types of textual instructions. To address the neglect of language instructions, we propose a novel framework that grounds the translation of visual information on instructions for language models. We introduce a mixture of visual projectors, each serving as a specialized visual-to-language translation expert based on the given instruction context to adapt to new tasks. To avoid using experts for irrelevant instruction contexts, we propose an expert recommendation strategy that reuses experts for tasks similar to those previously learned. Additionally, we introduce expert pruning to alleviate interference from the use of experts that cumulatively activated in previous tasks. Extensive experiments on diverse vision-language tasks demonstrate that our method outperforms existing continual learning approaches by generating instruction-following responses.