Multi-Modal Representation Learning via Semi-Supervised Rate Reduction for Generalized Category Discovery

TL;DR

This paper introduces SSR$^2$-GCD, a multi-modal learning framework for generalized category discovery that emphasizes intra-modality alignment and leverages vision-language models for improved recognition of known and unknown categories.

Contribution

The paper proposes a novel semi-supervised rate reduction method for multi-modal GCD, emphasizing intra-modality alignment and integrating vision-language prompts for enhanced knowledge transfer.

Findings

Outperforms existing methods on benchmark datasets.

Effectively learns cross-modality representations with desired structural properties.

Leverages vision-language models to improve recognition of unknown categories.

Abstract

Generalized Category Discovery (GCD) aims to identify both known and unknown categories, with only partial labels given for the known categories, posing a challenging open-set recognition problem. State-of-the-art approaches for GCD task are usually built on multi-modality representation learning, which is heavily dependent upon inter-modality alignment. However, few of them cast a proper intra-modality alignment to generate a desired underlying structure of representation distributions. In this paper, we propose a novel and effective multi-modal representation learning framework for GCD via Semi-Supervised Rate Reduction, called SSR$^2$-GCD, to learn cross-modality representations with desired structural properties based on emphasizing to properly align intra-modality relationships. Moreover, to boost knowledge transfer, we integrate prompt candidates by leveraging the inter-modal alignment offered by Vision Language Models. We conduct extensive experiments on generic and fine-grained benchmark datasets demonstrating superior performance of our approach.