Ask the Expert: Collaborative Inference for Vision Transformers with Near-Edge Accelerators

TL;DR

This paper introduces a collaborative inference framework for Vision Transformers that balances edge and near-edge processing, improving accuracy, reducing latency, and saving energy through dynamic expert selection and specialized training.

Contribution

It presents a novel routing mechanism and progressive training strategy for efficient, accurate Vision Transformer inference on edge devices with near-edge accelerators.

Findings

Expert accuracy improved by 4.12% on target subsets.

Overall accuracy increased by 2.76%.

Latency reduced by up to 45%. and energy consumption by up to 46%.

Abstract

Deploying Vision Transformers on edge devices is challenging due to their high computational complexity, while full offloading to cloud resources presents significant latency overheads. We propose a novel collaborative inference framework, which orchestrates a lightweight generalist ViT on an edge device and multiple medium-sized expert ViTs on a near-edge accelerator. A novel routing mechanism uses the edge model's Top-$\mathit{k}$ predictions to dynamically select the most relevant expert for samples with low confidence. We further design a progressive specialist training strategy to enhance expert accuracy on dataset subsets. Extensive experiments on the CIFAR-100 dataset using a real-world edge and near-edge testbed demonstrate the superiority of our framework. Specifically, the proposed training strategy improves expert specialization accuracy by 4.12% on target subsets and enhances overall accuracy by 2.76% over static experts. Moreover, our method reduces latency by up to 45% compared to edge execution, and energy consumption by up to 46% compared to just near-edge offload.