Towards Recognizing Unseen Categories in Unseen Domains

TL;DR

This paper introduces CuMix, a novel algorithm that jointly addresses zero-shot learning and domain generalization by simulating unseen domain and category shifts through curriculum-based mixup techniques, improving recognition of unseen visual concepts.

Contribution

The work presents CuMix, a unified approach that combines zero-shot learning and domain generalization by curriculum-driven mixup to handle unseen categories in unseen domains.

Findings

Effective on standard SL and DG datasets

Improves recognition of unseen categories in unseen domains

Demonstrates strong performance on DomainNet benchmark

Abstract

Current deep visual recognition systems suffer from severe performance degradation when they encounter new images from classes and scenarios unseen during training. Hence, the core challenge of Zero-Shot Learning (ZSL) is to cope with the semantic-shift whereas the main challenge of Domain Adaptation and Domain Generalization (DG) is the domain-shift. While historically ZSL and DG tasks are tackled in isolation, this work develops with the ambitious goal of solving them jointly,i.e. by recognizing unseen visual concepts in unseen domains. We presentCuMix (CurriculumMixup for recognizing unseen categories in unseen domains), a holistic algorithm to tackle ZSL, DG and ZSL+DG. The key idea of CuMix is to simulate the test-time domain and semantic shift using images and features from unseen domains and categories generated by mixing up the multiple source domains and categories available during training. Moreover, a curriculum-based mixing policy is devised to generate increasingly complex training samples. Results on standard SL and DG datasets and on ZSL+DG using the DomainNet benchmark demonstrate the effectiveness of our approach.