Dynamic Knowledge Distillation for Black-box Hypothesis Transfer Learning

TL;DR

This paper introduces a novel dynamic knowledge distillation method for hypothesis transfer learning that effectively leverages black-box source models without access to source data, demonstrated on benchmark and healthcare datasets.

Contribution

The paper proposes a new algorithm called dkdHTL that adaptively transfers knowledge from black-box models using instance-wise weighting, addressing a key challenge in hypothesis transfer learning.

Findings

Effective transfer of knowledge from black-box models demonstrated.

Improved performance on benchmark and healthcare datasets.

Adaptive weighting mechanism enhances transfer learning accuracy.

Abstract

In real world applications like healthcare, it is usually difficult to build a machine learning prediction model that works universally well across different institutions. At the same time, the available model is often proprietary, i.e., neither the model parameter nor the data set used for model training is accessible. In consequence, leveraging the knowledge hidden in the available model (aka. the hypothesis) and adapting it to a local data set becomes extremely challenging. Motivated by this situation, in this paper we aim to address such a specific case within the hypothesis transfer learning framework, in which 1) the source hypothesis is a black-box model and 2) the source domain data is unavailable. In particular, we introduce a novel algorithm called dynamic knowledge distillation for hypothesis transfer learning (dkdHTL). In this method, we use knowledge distillation with instance-wise weighting mechanism to adaptively transfer the "dark" knowledge from the source hypothesis to the target domain.The weighting coefficients of the distillation loss and the standard loss are determined by the consistency between the predicted probability of the source hypothesis and the target ground-truth label.Empirical results on both transfer learning benchmark datasets and a healthcare dataset demonstrate the effectiveness of our method.