DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks

TL;DR

The paper introduces DELTA, a novel transfer learning framework that preserves outer layer feature maps using attention mechanisms, leading to improved accuracy over existing methods.

Contribution

DELTA is a new regularized transfer learning approach that aligns outer layer feature maps with attention, enhancing performance on target tasks.

Findings

DELTA outperforms L2 and L2-SP in accuracy on new tasks.

Attention-based feature map alignment improves transfer learning.

DELTA effectively preserves target network outputs during fine-tuning.

Abstract

Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task. To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied. In this paper, we propose a novel regularized transfer learning framework DELTA, namely DEep Learning Transfer using Feature Map with Attention. Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network. Specifically, in addition to minimizing the empirical loss, DELTA intends to align the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in an supervised learning manner. We evaluate DELTA with the state-of-the-art algorithms, including L2 and L2-SP. The experiment results show that our proposed method outperforms these baselines with higher accuracy for new tasks.