Generate, Refine, and Encode: Leveraging Synthesized Novel Samples for On-the-Fly Fine-Grained Category Discovery

TL;DR

This paper introduces DiffGRE, a diffusion-based framework for on-the-fly fine-grained category discovery that synthesizes, refines, and encodes novel samples to improve recognition of known and unknown categories.

Contribution

The paper proposes a novel multi-stage diffusion-based framework, DiffGRE, combining generation, refinement, and encoding to enhance fine-grained category discovery with limited labeled data.

Findings

DiffGRE outperforms previous methods on six fine-grained datasets.

Synthesized samples improve the discovery of both known and unknown categories.

Refinement effectively filters diverse novel samples for training.

Abstract

In this paper, we investigate a practical yet challenging task: On-the-fly Category Discovery (OCD). This task focuses on the online identification of newly arriving stream data that may belong to both known and unknown categories, utilizing the category knowledge from only labeled data. Existing OCD methods are devoted to fully mining transferable knowledge from only labeled data. However, the transferability learned by these methods is limited because the knowledge contained in known categories is often insufficient, especially when few annotated data/categories are available in fine-grained recognition. To mitigate this limitation, we propose a diffusion-based OCD framework, dubbed DiffGRE, which integrates Generation, Refinement, and Encoding in a multi-stage fashion. Specifically, we first design an attribute-composition generation method based on cross-image interpolation in the diffusion latent space to synthesize novel samples. Then, we propose a diversity-driven refinement approach to select the synthesized images that differ from known categories for subsequent OCD model training. Finally, we leverage a semi-supervised leader encoding to inject additional category knowledge contained in synthesized data into the OCD models, which can benefit the discovery of both known and unknown categories during the on-the-fly inference process. Extensive experiments demonstrate the superiority of our DiffGRE over previous methods on six fine-grained datasets.