Prompt-Based Continual Compositional Zero-Shot Learning

TL;DR

This paper introduces PromptCCZSL, a novel framework for continual compositional zero-shot learning that leverages prompt-based methods and multi-teacher distillation to adapt to new attribute-object compositions while retaining prior knowledge.

Contribution

It proposes the first prompt-based continual learning approach for compositional zero-shot learning, incorporating session-aware prompts, multiple loss functions, and a new evaluation protocol.

Findings

PromptCCZSL outperforms prior methods on UT-Zappos and C-GQA benchmarks.

The framework effectively balances knowledge retention and adaptation.

Experimental results show significant improvements in compositional generalization.

Abstract

We tackle continual adaptation of vision-language models to new attributes, objects, and their compositions in Compositional Zero-Shot Learning (CZSL), while preventing forgetting of prior knowledge. Unlike classical continual learning where classes are disjoint, CCZSL is more complex as attributes and objects may reoccur across sessions while compositions remain unique. Built on a frozen VLM backbone, we propose the first Prompt-based Continual Compositional Zero-Shot Learning (PromptCCZSL) framework that retains prior knowledge through recency-weighted multi-teacher distillation. It employs session-aware compositional prompts to fuse multimodal features for new compositions, while attribute and object prompts are learned through session-agnostic fusion to maintain global semantic consistency, which is further stabilized by a Cosine Anchor Loss (CAL) to preserve prior knowledge. To enhance adaptation in the current session, an Orthogonal Projection Loss (OPL) ensures that new attribute and object embeddings remain distinct from previous ones, preventing overlap, while an Intra-Session Diversity Loss (IDL) promotes variation among current-session embeddings for richer, more discriminative representations. We also introduce a comprehensive protocol that jointly measures catastrophic forgetting and compositional generalization. Extensive experiments on UT-Zappos and C-GQA benchmarks demonstrate that PromptCCZSL achieves substantial improvements over prior VLM-based and non-VLM baselines, setting a new benchmark for CCZSL in closed-world settings.