Hierarchical Semantic Tree Anchoring for CLIP-Based Class-Incremental Learning

TL;DR

This paper introduces HASTEN, a hierarchical semantic tree anchoring method for CLIP-based class-incremental learning, which preserves hierarchical relationships to reduce forgetting and improve continual learning performance.

Contribution

HASTEN leverages external knowledge graphs and hyperbolic embedding to explicitly encode hierarchy, and employs gradient projection to prevent forgetting in CLIP-based CIL.

Findings

HASTEN outperforms existing CIL methods in experiments.

It effectively preserves hierarchical relationships during incremental learning.

The approach reduces catastrophic forgetting significantly.

Abstract

Class-Incremental Learning (CIL) enables models to learn new classes continually while preserving past knowledge. Recently, vision-language models like CLIP offer transferable features via multi-modal pre-training, making them well-suited for CIL. However, real-world visual and linguistic concepts are inherently hierarchical: a textual concept like "dog" subsumes fine-grained categories such as "Labrador" and "Golden Retriever," and each category entails its images. But existing CLIP-based CIL methods fail to explicitly capture this inherent hierarchy, leading to fine-grained class features drift during incremental updates and ultimately to catastrophic forgetting. To address this challenge, we propose HASTEN (Hierarchical Semantic Tree Anchoring) that anchors hierarchical information into CIL to reduce catastrophic forgetting. First, we employ an external knowledge graph as supervision to embed visual and textual features in hyperbolic space, effectively preserving hierarchical structure as data evolves. Second, to mitigate catastrophic forgetting, we project gradients onto the null space of the shared hyperbolic mapper, preventing interference with prior tasks. These two steps work synergistically to enable the model to resist forgetting by maintaining hierarchical relationships. Extensive experiments show that HASTEN consistently outperforms existing methods while providing a unified structured representation.