Does Optimal Source Task Performance Imply Optimal Pre-training for a Target Task?

TL;DR

Pre-training a neural network to optimal performance on a source task does not necessarily lead to the best transfer learning results; stopping earlier can sometimes yield better fine-tuning outcomes.

Contribution

This paper challenges the assumption that optimal source task performance is ideal for transfer learning, showing that earlier stopping can improve fine-tuning success.

Findings

Stopping source training early can enhance transfer learning.

Optimal source performance does not guarantee best target task results.

Learning ability diminishes with prolonged source training.

Abstract

Fine-tuning of pre-trained deep nets is commonly used to improve accuracies and training times for neural nets. It is generally assumed that pre-training a net for optimal source task performance best prepares it for fine-tuning to learn an arbitrary target task. This is generally not true. Stopping source task training, prior to optimal performance, can create a pre-trained net better suited for fine-tuning to learn a new task. We perform several experiments demonstrating this effect, as well as the influence of the amount of training and of learning rate. Additionally, our results indicate that this reflects a general loss of learning ability that even extends to relearning the source task.