Generalized Category Discovery under Domain Shifts: From Vision to Vision-Language Models

TL;DR

This paper introduces three frameworks for generalized category discovery under domain shifts, leveraging foundation models from vision to vision-language, with extensive experiments showing improved performance.

Contribution

It proposes novel methods HiLo, HLPrompt, and VLPrompt that adapt foundation models for GCD across domain shifts, a scenario less explored in prior work.

Findings

Consistent improvements over strong baselines on synthetic and real-world shifts.

The methods effectively disentangle domain and semantic features.

Leveraging vision-language models enhances GCD performance.

Abstract

Generalized Category Discovery (GCD) aims to categorize unlabelled instances from both known and unknown classes by transferring knowledge from labelled data of known classes. Existing methods assume all data comes from a single domain, yet real-world unlabelled data often exhibits domain shifts alongside semantic shifts. We study GCD under domain shifts and propose three frameworks that adapt foundation models, ranging from self-supervised vision models to vision-language models. (i) HiLo disentangles domain and semantic features through multi-level feature extraction and mutual information minimization, combined with PatchMix augmentation and curriculum sampling. (ii) HLPrompt extends HiLo with semantic-aware spatial prompt tuning to suppress background and domain noise. (iii) VLPrompt leverages vision-language models via factorized textual prompts and cross-modal consistency regularization. The three methods share core design principles while operating on different foundation backbones, making them suitable for different deployment scenarios. Extensive experiments on synthetic corruptions and real-world multi-domain shifts demonstrate consistent improvements over strong baselines. Project page: https://visual-ai.github.io/hilo/