InstantFT: An FPGA-Based Runtime Subsecond Fine-tuning of CNN Models

TL;DR

InstantFT is an FPGA-based method enabling ultra-fast, energy-efficient CNN fine-tuning on IoT devices, achieving subsecond adaptation times and comparable accuracy to existing approaches.

Contribution

The paper introduces InstantFT, a novel FPGA-based approach that significantly accelerates CNN fine-tuning for resource-limited IoT platforms using optimized PEFT techniques.

Findings

Fine-tunes CNN 17.4x faster than LoRA-based methods

Reduces fine-tuning time to 0.36 seconds

Improves energy efficiency by 16.3x

Abstract

Training deep neural networks (DNNs) requires significantly more computation and memory than inference, making runtime adaptation of DNNs challenging on resource-limited IoT platforms. We propose InstantFT, an FPGA-based method for ultra-fast CNN fine-tuning on IoT devices, by optimizing the forward and backward computations in parameter-efficient fine-tuning (PEFT). Experiments on datasets with concept drift demonstrate that InstantFT fine-tunes a pre-trained CNN 17.4x faster than existing Low-Rank Adaptation (LoRA)-based approaches, while achieving comparable accuracy. Our FPGA-based InstantFT reduces the fine-tuning time to just 0.36s and improves energy-efficiency by 16.3x, enabling on-the-fly adaptation of CNNs to non-stationary data distributions.