Skip2-LoRA: A Lightweight On-device DNN Fine-tuning Method for Low-cost Edge Devices

TL;DR

Skip2-LoRA introduces a lightweight, efficient fine-tuning approach for deep neural networks on low-cost edge devices, significantly reducing training time while maintaining accuracy.

Contribution

It proposes a novel architecture combining LoRA adapters with caching to enable fast on-device fine-tuning for edge AI applications.

Findings

Reduces fine-tuning time by 90% on average

Maintains accuracy comparable to full fine-tuning

Operates efficiently on microcontroller hardware

Abstract

This paper proposes Skip2-LoRA as a lightweight fine-tuning method for deep neural networks to address the gap between pre-trained and deployed models. In our approach, trainable LoRA (low-rank adaptation) adapters are inserted between the last layer and every other layer to enhance the network expressive power while keeping the backward computation cost low. This architecture is well-suited to cache intermediate computation results of the forward pass and then can skip the forward computation of seen samples as training epochs progress. We implemented the combination of the proposed architecture and cache, denoted as Skip2-LoRA, and tested it on a $15 single board computer. Our results show that Skip2-LoRA reduces the fine-tuning time by 90.0% on average compared to the counterpart that has the same number of trainable parameters while preserving the accuracy, while taking only a few seconds on the microcontroller board.