Expand Heterogeneous Learning Systems with Selective Multi-Source Knowledge Fusion

TL;DR

This paper introduces HaT, a framework that selectively fuses multiple models' knowledge to expand learning systems for new domains, improving accuracy and reducing communication costs.

Contribution

The paper proposes a novel selective multi-source knowledge fusion framework, HaT, addressing data heterogeneity and reliability issues in expanding learning systems.

Findings

HaT outperforms state-of-the-art methods by up to 16.5% accuracy.

HaT reduces communication traffic by up to 39%.

Effective knowledge selection and fusion improve model expansion quality.

Abstract

Expanding existing learning systems to provide high-quality customized models for more domains, such as new users, is challenged by the limited labeled data and the data and device heterogeneities. While knowledge distillation methods could overcome label scarcity and device heterogeneity, they assume the teachers are fully reliable and overlook the data heterogeneity, which prevents the direct adoption of existing models. To address this problem, this paper proposes a framework, HaT, to expand learning systems. It first selects multiple high-quality models from the system at a low cost and then fuses their knowledge by assigning sample-wise weights to their predictions. Later, the fused knowledge is selectively injected into the customized models based on the knowledge quality. Extensive experiments on different tasks, modalities, and settings show that HaT outperforms state-of-the-art baselines by up to 16.5% accuracy and saves up to 39% communication traffic.