Efficient Few-Shot Learning for Edge AI via Knowledge Distillation on MobileViT

TL;DR

This paper introduces a knowledge distillation approach for MobileViT to enhance few-shot learning on edge devices, achieving higher accuracy with significantly reduced model size and energy consumption.

Contribution

It presents a novel pre-training method using knowledge distillation for MobileViT, improving few-shot classification accuracy and efficiency on edge hardware.

Findings

Achieved 14% and 6.7% accuracy improvements for one-shot and five-shot tasks.

Reduced model parameters by 69% and FLOPs by 88%.

Lowered power consumption by 37% on Jetson Orin Nano.

Abstract

Efficient and adaptable deep learning models are an important area of deep learning research, driven by the need for highly efficient models on edge devices. Few-shot learning enables the use of deep learning models in low-data regimes, a capability that is highly sought after in real-world applications where collecting large annotated datasets is costly or impractical. This challenge is particularly relevant in edge scenarios, where connectivity may be limited, low-latency responses are required, or energy consumption constraints are critical. We propose and evaluate a pre-training method for the MobileViT backbone designed for edge computing. Specifically, we employ knowledge distillation, which transfers the generalization ability of a large-scale teacher model to a lightweight student model. This method achieves accuracy improvements of 14% and 6.7% for one-shot and five-shot classification, respectively, on the MiniImageNet benchmark, compared to the ResNet12 baseline, while reducing by 69% the number of parameters and by 88% the computational complexity of the model, in FLOPs. Furthermore, we deployed the proposed models on a Jetson Orin Nano platform and measured power consumption directly at the power supply, showing that the dynamic energy consumption is reduced by 37% with a latency of 2.6 ms. These results demonstrate that the proposed method is a promising and practical solution for deploying few-shot learning models on edge AI hardware.