Gradual Tuning: a better way of Fine Tuning the parameters of a Deep Neural Network

TL;DR

Gradual Tuning is a novel fine-tuning method that progressively adjusts more network parameters, reducing catastrophic forgetting and maintaining or improving performance on new tasks.

Contribution

The paper introduces Gradual Tuning, an alternative fine-tuning strategy that incrementally modifies network parameters to better preserve prior knowledge during transfer learning.

Findings

Reduces catastrophic forgetting compared to standard fine-tuning.

Maintains or improves performance on the new task.

Effective across different network sizes and regularization techniques.

Abstract

In this paper we present an alternative strategy for fine-tuning the parameters of a network. We named the technique Gradual Tuning. Once trained on a first task, the network is fine-tuned on a second task by modifying a progressively larger set of the network's parameters. We test Gradual Tuning on different transfer learning tasks, using networks of different sizes trained with different regularization techniques. The result shows that compared to the usual fine tuning, our approach significantly reduces catastrophic forgetting of the initial task, while still retaining comparable if not better performance on the new task.