CoQMoE: Co-Designed Quantization and Computation Orchestration for Mixture-of-Experts Vision Transformer on FPGA

TL;DR

This paper introduces a novel FPGA accelerator for quantized Mixture-of-Experts Vision Transformers, achieving high throughput and energy efficiency with minimal accuracy loss through innovative quantization and resource-aware design.

Contribution

It proposes a dual-stage quantization scheme and a resource-aware FPGA architecture tailored for MoE-ViTs, enabling efficient deployment with high performance and low energy consumption.

Findings

Achieves 155 fps throughput on FPGA.

5.35× throughput improvement over SOTA.

Over 80% energy reduction with <1% accuracy loss.

Abstract

Vision Transformers (ViTs) exhibit superior performance in computer vision tasks but face deployment challenges on resource-constrained devices due to high computational/memory demands. While Mixture-of-Experts Vision Transformers (MoE-ViTs) mitigate this through a scalable architecture with sub-linear computational growth, their hardware implementation on FPGAs remains constrained by resource limitations. This paper proposes a novel accelerator for efficiently implementing quantized MoE models on FPGAs through two key innovations: (1) A dual-stage quantization scheme combining precision-preserving complex quantizers with hardware-friendly simplified quantizers via scale reparameterization, with only 0.28 $\%$ accuracy loss compared to full precision; (2) A resource-aware accelerator architecture featuring latency-optimized streaming attention kernels and reusable linear operators, effectively balancing performance and resource consumption. Experimental results demonstrate that our accelerator achieves nearly 155 frames per second, a 5.35$\times$ improvement in throughput, and over $80\%$ energy reduction compared to state-of-the-art (SOTA) FPGA MoE accelerators, while maintaining $<1\%$ accuracy loss across vision benchmarks. Our implementation is available at https://github.com/DJ000011/CoQMoE.