Design Rules for Extreme-Edge Scientific Computing on AI Engines

TL;DR

This paper evaluates the use of AI Engines versus programmable logic for extreme-edge scientific neural networks, providing architectural insights, optimization strategies, and demonstrating successful deployments on FPGA SoCs.

Contribution

It introduces the LARE metric for comparing AI Engines and programmable logic, and offers dataflow optimizations for low-latency scientific inference.

Findings

AI Engines outperform programmable logic for larger models based on LARE metric.

Spatial and API-level dataflow optimizations improve inference latency.

End-to-end neural networks that don't fit on programmable logic can be deployed on AI Engines.

Abstract

Extreme-edge scientific applications use machine learning models to analyze sensor data and make real-time decisions. Their stringent latency and throughput requirements demand small batch sizes and require that model weights remain fully on-chip. Spatial dataflow implementations are common for extreme-edge applications. Spatial dataflow works well for small networks, but it fails to scale to larger models due to inherent resource scaling limitations. AI Engines on modern FPGA SoCs offer a promising alternative with high compute density and additional on-chip memory. However, the architecture, programming model, and performance-scaling behavior of AI Engines differ fundamentally from those of the programmable logic, making direct comparison non-trivial and the benefits of using AI Engines unclear. This work addresses how and when extreme-edge scientific neural networks should be implemented on AI Engines versus programmable logic. We provide systematic architectural characterization and micro-benchmarking and introduce a latency-adjusted resource equivalence (LARE) metric that identifies when AI Engine implementations outperform programmable logic designs. We further propose spatial and API-level dataflow optimizations tailored to low-latency scientific inference. Finally, we demonstrate the successful deployment of end-to-end neural networks on AI Engines that cannot fit on programmable logic when using the hlsml toolchain.