TOMCAT: Test-time Comprehensive Knowledge Accumulation for Compositional Zero-Shot Learning

TL;DR

TOMCAT introduces a test-time knowledge accumulation method for compositional zero-shot learning, leveraging multimodal data and adaptive updates to improve recognition of unseen attribute-object pairs.

Contribution

The paper proposes a novel test-time approach that accumulates multimodal knowledge and adaptively updates prototypes to handle distribution shifts in CZSL.

Findings

Achieves state-of-the-art results on four benchmark datasets.

Effective in both closed-world and open-world settings.

Demonstrates robustness to distribution shifts during testing.

Abstract

Compositional Zero-Shot Learning (CZSL) aims to recognize novel attribute-object compositions based on the knowledge learned from seen ones. Existing methods suffer from performance degradation caused by the distribution shift of label space at test time, which stems from the inclusion of unseen compositions recombined from attributes and objects. To overcome the challenge, we propose a novel approach that accumulates comprehensive knowledge in both textual and visual modalities from unsupervised data to update multimodal prototypes at test time. Building on this, we further design an adaptive update weight to control the degree of prototype adjustment, enabling the model to flexibly adapt to distribution shift during testing. Moreover, a dynamic priority queue is introduced that stores high-confidence images to acquire visual knowledge from historical images for inference. Considering the semantic consistency of multimodal knowledge, we align textual and visual prototypes by multimodal collaborative representation learning. Extensive experiments indicate that our approach achieves state-of-the-art performance on four benchmark datasets under both closed-world and open-world settings. Code will be available at https://github.com/xud-yan/TOMCAT .