TRINE: A Token-Aware, Runtime-Adaptive FPGA Inference Engine for Multimodal AI

TL;DR

TRINE is a versatile FPGA inference engine that adaptively executes multimodal AI models with high efficiency, low latency, and minimal reconfiguration, enabling real-time processing across vision, language, and graph tasks.

Contribution

It introduces a unified, runtime-adaptive FPGA accelerator capable of executing diverse multimodal models without reconfiguration, using a mode-switchable engine and novel layer offloading techniques.

Findings

Reduces latency by up to 22.57x compared to RTX 4090.

Achieves up to 6.86x latency reduction over Jetson Orin Nano at similar power.

Token pruning and dependency-aware layer offloading significantly improve throughput.

Abstract

Multimodal stacks that mix ViTs, CNNs, GNNs, and transformer NLP strain embedded platforms because their compute/memory patterns diverge and hard real-time targets leave little slack. TRINE is a single-bitstream FPGA accelerator and compiler that executes end-to-end multimodal inference without reconfiguration. Layers are unified as DDMM/SDDMM/SpMM and mapped to a mode-switchable engine that toggles at runtime among weight/output-stationary systolic, 1xCS SIMD, and a routable adder tree (RADT) on a shared PE array. A width-matched, two-stage top-k unit enables in-stream token pruning, while dependency-aware layer offloading (DALO) overlaps independent kernels across reconfigurable processing units to sustain utilization. Evaluated on Alveo U50 and ZCU104, TRINE reduces latency by up to 22.57x vs. RTX 4090 and 6.86x vs. Jetson Orin Nano at 20-21 W; token pruning alone yields up to 7.8x on ViT-heavy pipelines, and DALO contributes up to 79% throughput improvement. With int8 quantization, accuracy drops remain <2.5% across representative tasks, delivering state-of-the-art latency and energy efficiency for unified vision, language, and graph workloads-in one bitstream.