Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering

TL;DR

This paper introduces QUAD, a novel continual VQA method that uses only past questions for regularization, reducing memory needs and privacy issues, while maintaining visual-linguistic knowledge across tasks.

Contribution

The paper proposes a question-only replay mechanism with attention distillation for continual VQA, addressing stability-plasticity trade-off without visual data storage.

Findings

QUAD outperforms existing methods on VQAv2 and NExT-QA datasets.

It effectively mitigates overfitting to current task answer spaces.

The approach reduces memory overhead and privacy concerns.

Abstract

Continual Learning in Visual Question Answering (VQACL) requires models to acquire new visual-linguistic skills (plasticity) while preserving previously learned knowledge (stability). The inherent multimodality of VQACL exacerbates this challenge, as models must balance stability across visual and textual domains while adapting to novel objects and reasoning tasks. Existing methods, primarily designed for unimodal settings, often fall short in addressing this dual requirement. In this work, we present QUestion-only replay with Attention Distillation (QUAD), a novel approach for VQACL that leverages only past task questions for regularization. By eliminating the need to store visual data, QUAD not only reduces memory overhead, but also alleviates privacy concerns. Our method introduces a Question-only Replay mechanism that selectively reuses prior task questions to counteract overfitting to the answer space of the current task, addressing the problem out of answer set. Complementing this, we propose Attention Consistency Distillation to enforce both intra-modal and inter-modal attention consistency across tasks, preserving essential visual-linguistic associations. Extensive experiments on VQAv2 and NExT-QA demonstrate that QUAD significantly outperforms state-of-the-art methods, achieving robust performance in continual VQA. Code is available at: https://github.com/IemProg/QUAD.