Efficient Few-Shot Continual Learning in Vision-Language Models

TL;DR

This paper introduces LoRSU, a method for efficient, structured updates to image encoders in vision-language models, enabling effective few-shot continual learning with significantly reduced computational costs.

Contribution

LoRSU is a novel approach that selectively updates critical parameters in image encoders, improving efficiency and performance in continual learning scenarios.

Findings

Reduces computational overhead by over 25x compared to full model updates.

Maintains high performance in few-shot continual learning tasks.

Demonstrates scalability and robustness across VQA benchmarks.

Abstract

Vision-language models (VLMs) excel in tasks such as visual question answering and image captioning. However, VLMs are often limited by their use of pretrained image encoders, like CLIP, leading to image understanding errors that hinder overall performance. On top of that, real-world applications often require the model to be continuously adapted as new and often limited data continuously arrive. To address this, we propose LoRSU (Low-Rank Adaptation with Structured Updates), a robust and computationally efficient method for selectively updating image encoders within VLMs. LoRSU introduces structured and localized parameter updates, effectively correcting performance on previously error-prone data while preserving the model's general robustness. Our approach leverages theoretical insights to identify and update only the most critical parameters, achieving significant resource efficiency. Specifically, we demonstrate that LoRSU reduces computational overhead by over 25x compared to full VLM updates, without sacrificing performance. Experimental results on VQA tasks in the few-shot continual learning setting, validate LoRSU's scalability, efficiency, and effectiveness, making it a compelling solution for image encoder adaptation in resource-constrained environments.