One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answering

TL;DR

This paper introduces GaB, a novel data-free continual learning approach for Visual Question Answering that generates pseudo-rehearsal data using a vision-language model, effectively mitigating forgetting without storing past data.

Contribution

GaB is the first data-free method leveraging VLMs for pseudo-rehearsal data generation in continual VQA, with a balancing module to improve data distribution alignment.

Findings

GaB outperforms all data-free baselines in continual VQA tasks.

GaB matches the performance of methods with access to past data.

The balancing module effectively aligns generated data with ground-truth distribution.

Abstract

Vision-Language Models (VLMs) have shown significant promise in Visual Question Answering (VQA) tasks by leveraging web-scale multimodal datasets. However, these models often struggle with continual learning due to catastrophic forgetting when adapting to new tasks. As an effective remedy to mitigate catastrophic forgetting, rehearsal strategy uses the data of past tasks upon learning new task. However, such strategy incurs the need of storing past data, which might not be feasible due to hardware constraints or privacy concerns. In this work, we propose the first data-free method that leverages the language generation capability of a VLM, instead of relying on external models, to produce pseudo-rehearsal data for addressing continual VQA. Our proposal, named as GaB, generates pseudo-rehearsal data by posing previous task questions on new task data. Yet, despite being effective, the distribution of generated questions skews towards the most frequently posed questions due to the limited and task-specific training data. To mitigate this issue, we introduce a pseudo-rehearsal balancing module that aligns the generated data towards the ground-truth data distribution using either the question meta-statistics or an unsupervised clustering method. We evaluate our proposed method on two recent benchmarks, \ie VQACL-VQAv2 and CLOVE-function benchmarks. GaB outperforms all the data-free baselines with substantial improvement in maintaining VQA performance across evolving tasks, while being on-par with methods with access to the past data.