TALON: Test-time Adaptive Learning for On-the-Fly Category Discovery

TL;DR

TALON introduces a test-time adaptive learning framework for on-the-fly category discovery that dynamically updates prototypes and encoders, significantly improving novel class recognition and reducing category fragmentation.

Contribution

It proposes a novel test-time adaptation approach with semantic-aware prototype updates and encoder refinement, addressing limitations of fixed feature quantization in OCD.

Findings

Outperforms hash-based state-of-the-art methods on OCD benchmarks.

Significantly improves accuracy on novel classes.

Reduces category explosion and pseudo-class fragmentation.

Abstract

On-the-fly category discovery (OCD) aims to recognize known categories while simultaneously discovering novel ones from an unlabeled online stream, using a model trained only on labeled data. Existing approaches freeze the feature extractor trained offline and employ a hash-based framework that quantizes features into binary codes as class prototypes. However, discovering novel categories with a fixed knowledge base is counterintuitive, as the learning potential of incoming data is entirely neglected. In addition, feature quantization introduces information loss, diminishes representational expressiveness, and amplifies intra-class variance. It often results in category explosion, where a single class is fragmented into multiple pseudo-classes. To overcome these limitations, we propose a test-time adaptation framework that enables learning through discovery. It incorporates two complementary strategies: a semantic-aware prototype update and a stable test-time encoder update. The former dynamically refines class prototypes to enhance classification, whereas the latter integrates new information directly into the parameter space. Together, these components allow the model to continuously expand its knowledge base with newly encountered samples. Furthermore, we introduce a margin-aware logit calibration in the offline stage to enlarge inter-class margins and improve intra-class compactness, thereby reserving embedding space for future class discovery. Experiments on standard OCD benchmarks demonstrate that our method substantially outperforms existing hash-based state-of-the-art approaches, yielding notable improvements in novel-class accuracy and effectively mitigating category explosion. The code is publicly available at \textcolor{blue}{https://github.com/ynanwu/TALON}.