Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning

TL;DR

This paper introduces DualOpt, a novel optimizer that decouples techniques for training neural networks from scratch and fine-tuning pre-trained models, enhancing convergence, generalization, and knowledge retention.

Contribution

The paper proposes DualOpt, which incorporates layer-wise weight decay for scratch training and weight rollback for fine-tuning, addressing the distinct needs of these paradigms.

Findings

DualOpt achieves state-of-the-art results across multiple tasks.

Layer-wise weight decay improves convergence and generalization.

Weight rollback mitigates knowledge forgetting during fine-tuning.

Abstract

With the accumulation of resources in the era of big data and the rise of pre-trained models in deep learning, optimizing neural networks for various tasks often involves different strategies for fine-tuning pre-trained models versus training from scratch. However, existing optimizers primarily focus on reducing the loss function by updating model parameters, without fully addressing the unique demands of these two major paradigms. In this paper, we propose DualOpt, a novel approach that decouples optimization techniques specifically tailored for these distinct training scenarios. For training from scratch, we introduce real-time layer-wise weight decay, designed to enhance both convergence and generalization by aligning with the characteristics of weight updates and network architecture. For more importantly fine-tuning, we integrate weight rollback with the optimizer, incorporating a rollback term into each weight update step. This ensures consistency in the weight distribution between upstream and downstream models, effectively mitigating knowledge forgetting and improving fine-tuning performance. Additionally, we extend the layer-wise weight decay to dynamically adjust the rollback levels across layers, adapting to the varying demands of different downstream tasks. Extensive experiments across diverse tasks, including image classification, object detection, semantic segmentation, and instance segmentation, demonstrate the broad applicability and state-of-the-art performance of DualOpt. Code is available at https://github.com/qklee-lz/OLOR-AAAI-2024.