Transfer Learning using Neural Ordinary Differential Equations

TL;DR

This paper explores using Neural Ordinary Differential Equations (NODE) for transfer learning, demonstrating improved stability and convergence when fine-tuning EfficientNets on CIFAR-10, with a trade-off between precision and speed.

Contribution

It introduces the novel application of NODE for transfer learning, showing enhanced stability and convergence during fine-tuning of deep models.

Findings

NODE provides more stable training and validation.

Neural ODEs enable a trade-off between numerical precision and computational speed.

Transfer learning with NODE results in stable loss convergence.

Abstract

A concept of using Neural Ordinary Differential Equations(NODE) for Transfer Learning has been introduced. In this paper we use the EfficientNets to explore transfer learning on CIFAR-10 dataset. We use NODE for fine-tuning our model. Using NODE for fine tuning provides more stability during training and validation.These continuous depth blocks can also have a trade off between numerical precision and speed .Using Neural ODEs for transfer learning has resulted in much stable convergence of the loss function.