Exploiting Task Relationships in Continual Learning via Transferability-Aware Task Embeddings

TL;DR

This paper introduces a transferability-aware task embedding called H-embedding, integrated into a hypernet framework, to improve continual learning by effectively leveraging inter-task relationships and enhancing transfer.

Contribution

It proposes a novel H-embedding derived from information theory, enabling online, low-dimensional, and efficient task-conditioned model learning for continual learning.

Findings

Outperforms baseline and SOTA methods on CIFAR-100, ImageNet-R, and DomainNet.

Efficiently captures intrinsic task relationships with low storage overhead.

Supports end-to-end training with practical computational requirements.

Abstract

Continual learning (CL) has been a critical topic in contemporary deep neural network applications, where higher levels of both forward and backward transfer are desirable for an effective CL performance. Existing CL strategies primarily focus on task models, either by regularizing model updates or by separating task-specific and shared components, while often overlooking the potential of leveraging inter-task relationships to enhance transfer. To address this gap, we propose a transferability-aware task embedding, termed H-embedding, and construct a hypernet framework under its guidance to learn task-conditioned model weights for CL tasks. Specifically, H-embedding is derived from an information theoretic measure of transferability and is designed to be online and easy to compute. Our method is also characterized by notable practicality, requiring only the storage of a low-dimensional task embedding per task and supporting efficient end-to-end training. Extensive evaluations on benchmarks including CIFAR-100, ImageNet-R, and DomainNet show that our framework performs prominently compared to various baseline and SOTA approaches, demonstrating strong potential in capturing and utilizing intrinsic task relationships. Our code is publicly available at https://github.com/viki760/Hembedding_Guided_Hypernet.