HAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learning

TL;DR

This paper introduces a novel lifelong heterogeneous learning setting and proposes HAD, a distillation method that effectively retains diverse knowledge across different task types, especially in dense prediction tasks.

Contribution

The paper formalizes lifelong heterogeneous learning and develops HAD, a new exemplar-free distillation approach tailored for preserving heterogeneous knowledge across diverse tasks.

Findings

HAD significantly outperforms existing methods in dense prediction lifelong learning scenarios.

The heterogeneity-aware distillation losses effectively address output imbalance and focus on informative pixels.

Extensive experiments validate HAD's effectiveness in preserving heterogeneous knowledge.

Abstract

Lifelong learning aims to preserve knowledge acquired from previous tasks while incorporating knowledge from a sequence of new tasks. However, most prior work explores only streams of homogeneous tasks (\textit{e.g.}, only classification tasks) and neglects the scenario of learning across heterogeneous tasks that possess different structures of outputs. In this work, we formalize this broader setting as lifelong heterogeneous learning (LHL). Departing from conventional lifelong learning, the task sequence of LHL spans different task types, and the learner needs to retain heterogeneous knowledge for different output space structures. To instantiate the LHL, we focus on LHL in the context of dense prediction (LHL4DP), a realistic and challenging scenario. To this end, we propose the Heterogeneity-Aware Distillation (HAD) method, an exemplar-free approach that preserves previously gained heterogeneous knowledge by self-distillation in each training phase. The proposed HAD comprises two complementary components, including a distribution-balanced heterogeneity-aware distillation loss to alleviate the global imbalance of prediction distribution and a salience-guided heterogeneity-aware distillation loss that concentrates learning on informative edge pixels extracted with the Sobel operator. Extensive experiments demonstrate that the proposed HAD method significantly outperforms existing methods in this new scenario.