Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Data

TL;DR

This paper introduces a new learning problem focused on adapting pre-trained models to target domains with partial label data, highlighting challenges and proposing solutions to preserve class accuracy and improve overall performance.

Contribution

It defines the problem of holistic transfer with partial target data, constructs benchmarks, and proposes methods to maintain accuracy of missing classes during adaptation.

Findings

Discovered the dilemma between domain adaptation and class preservation.

Proposed solutions effectively maintain missing class accuracy.

Established baseline methods for holistic transfer with partial data.

Abstract

We propose a learning problem involving adapting a pre-trained source model to the target domain for classifying all classes that appeared in the source data, using target data that covers only a partial label space. This problem is practical, as it is unrealistic for the target end-users to collect data for all classes prior to adaptation. However, it has received limited attention in the literature. To shed light on this issue, we construct benchmark datasets and conduct extensive experiments to uncover the inherent challenges. We found a dilemma -- on the one hand, adapting to the new target domain is important to claim better performance; on the other hand, we observe that preserving the classification accuracy of classes missing in the target adaptation data is highly challenging, let alone improving them. To tackle this, we identify two key directions: 1) disentangling domain gradients from classification gradients, and 2) preserving class relationships. We present several effective solutions that maintain the accuracy of the missing classes and enhance the overall performance, establishing solid baselines for holistic transfer of pre-trained models with partial target data.