Distributionally Robust Transfer Learning

TL;DR

This paper introduces TransDRO, a novel transfer learning method that optimizes for worst-case scenarios across diverse source distributions, improving robustness and accuracy especially with limited target data.

Contribution

The paper proposes TransDRO, a distributionally robust transfer learning approach that relaxes similarity constraints and provides theoretical guarantees and practical benefits.

Findings

TransDRO achieves faster convergence than target-only models.

It demonstrates robustness and accuracy in electronic health records data.

The method bridges transfer learning and distributional robustness.

Abstract

Many existing transfer learning methods rely on leveraging information from source data that closely resembles the target data. However, this approach often overlooks valuable knowledge that may be present in different yet potentially related auxiliary samples. When dealing with a limited amount of target data and a diverse range of source models, our paper introduces a novel approach, Distributionally Robust Optimization for Transfer Learning (TransDRO), that breaks free from strict similarity constraints. TransDRO is designed to optimize the most adversarial loss within an uncertainty set, defined as a collection of target populations generated as a convex combination of source distributions that guarantee excellent prediction performances for the target data. TransDRO effectively bridges the realms of transfer learning and distributional robustness prediction models. We establish the identifiability of TransDRO and its interpretation as a weighted average of source models closest to the baseline model. We also show that TransDRO achieves a faster convergence rate than the model fitted with the target data. Our comprehensive numerical studies and analysis of multi-institutional electronic health records data using TransDRO further substantiate the robustness and accuracy of TransDRO, highlighting its potential as a powerful tool in transfer learning applications.