Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning

TL;DR

This paper introduces a novel continual learning approach that leverages semantic knowledge from pre-trained text-encoders, specifically using CLIP, to improve knowledge retention and transfer across tasks by integrating semantic guidance and knowledge distillation.

Contribution

It proposes Semantically-guided Representation Learning and Knowledge Distillation modules that utilize text embeddings for enhanced continual learning performance.

Findings

Outperforms existing methods on general datasets

Effective in fine-grained classification tasks

Demonstrates improved knowledge transfer and retention

Abstract

Deep neural networks (DNNs) excel on fixed datasets but struggle with incremental and shifting data in real-world scenarios. Continual learning addresses this challenge by allowing models to learn from new data while retaining previously learned knowledge. Existing methods mainly rely on visual features, often neglecting the rich semantic information encoded in text. The semantic knowledge available in the label information of the images, offers important semantic information that can be related with previously acquired knowledge of semantic classes. Consequently, effectively leveraging this information throughout continual learning is expected to be beneficial. To address this, we propose integrating semantic guidance within and across tasks by capturing semantic similarity using text embeddings. We start from a pre-trained CLIP model, employ the \emph{Semantically-guided Representation Learning (SG-RL)} module for a soft-assignment towards all current task classes, and use the Semantically-guided Knowledge Distillation (SG-KD) module for enhanced knowledge transfer. Experimental results demonstrate the superiority of our method on general and fine-grained datasets. Our code can be found in https://github.com/aprilsveryown/semantically-guided-continual-learning.