OmniGCD: Abstracting Generalized Category Discovery for Modality Agnosticism

TL;DR

OmniGCD introduces a modality-agnostic approach for generalized category discovery that operates without dataset-specific fine-tuning, leveraging synthetic training and a Transformer-based model to improve zero-shot classification across multiple modalities.

Contribution

It proposes a novel modality-agnostic GCD method trained once on synthetic data, enabling zero-shot discovery across diverse datasets and modalities without fine-tuning.

Findings

Improves classification accuracy for known and novel classes across four modalities.

Performs zero-shot GCD on 16 datasets without dataset-specific fine-tuning.

Highlights the importance of strong encoders and decoupling representation learning from category discovery.

Abstract

Generalized Category Discovery (GCD) challenges methods to identify known and novel classes using partially labeled data, mirroring human category learning. Unlike prior GCD methods, which operate within a single modality and require dataset-specific fine-tuning, we propose a modality-agnostic GCD approach inspired by the human brain's abstract category formation. Our $\textbf{OmniGCD}$ leverages modality-specific encoders (e.g., vision, audio, text, remote sensing) to process inputs, followed by dimension reduction to construct a $\textbf{GCD latent space}$, which is transformed at test-time into a representation better suited for clustering using a novel synthetically trained Transformer-based model. To evaluate OmniGCD, we introduce a $\textbf{zero-shot GCD setting}$ where no dataset-specific fine-tuning is allowed, enabling modality-agnostic category discovery. $\textbf{Trained once on synthetic data}$, OmniGCD performs zero-shot GCD across 16 datasets spanning four modalities, improving classification accuracy for known and novel classes over baselines (average percentage point improvement of $\textbf{+6.2}$, $\textbf{+17.9}$, $\textbf{+1.5}$ and $\textbf{+12.7}$ for vision, text, audio and remote sensing). This highlights the importance of strong encoders while decoupling representation learning from category discovery. Improving modality-agnostic methods will propagate across modalities, enabling encoder development independent of GCD. Our work serves as a benchmark for future modality-agnostic GCD works, paving the way for scalable, human-inspired category discovery. All code is available $\href{https://github.com/Jordan-HS/OmniGCD}{here}$