MoP-CLIP: A Mixture of Prompt-Tuned CLIP Models for Domain Incremental Learning

TL;DR

MoP-CLIP introduces a mixture of prompt-tuned CLIP models that effectively handle domain shifts in incremental learning, outperforming existing methods especially in out-of-distribution scenarios.

Contribution

This paper proposes MoP-CLIP, a novel domain incremental learning approach using prompt-tuned CLIP models to adapt to both known and unseen domains at inference.

Findings

MoP-CLIP performs competitively in standard DIL settings.

MoP-CLIP outperforms state-of-the-art methods under domain shift.

Existing DIL methods struggle with out-of-distribution data.

Abstract

Despite the recent progress in incremental learning, addressing catastrophic forgetting under distributional drift is still an open and important problem. Indeed, while state-of-the-art domain incremental learning (DIL) methods perform satisfactorily within known domains, their performance largely degrades in the presence of novel domains. This limitation hampers their generalizability, and restricts their scalability to more realistic settings where train and test data are drawn from different distributions. To address these limitations, we present a novel DIL approach based on a mixture of prompt-tuned CLIP models (MoP-CLIP), which generalizes the paradigm of S-Prompting to handle both in-distribution and out-of-distribution data at inference. In particular, at the training stage we model the features distribution of every class in each domain, learning individual text and visual prompts to adapt to a given domain. At inference, the learned distributions allow us to identify whether a given test sample belongs to a known domain, selecting the correct prompt for the classification task, or from an unseen domain, leveraging a mixture of the prompt-tuned CLIP models. Our empirical evaluation reveals the poor performance of existing DIL methods under domain shift, and suggests that the proposed MoP-CLIP performs competitively in the standard DIL settings while outperforming state-of-the-art methods in OOD scenarios. These results demonstrate the superiority of MoP-CLIP, offering a robust and general solution to the problem of domain incremental learning.