Work-in-Progress: Real-Time Neural Network Inference on a Custom RISC-V Multicore Vector Processor

TL;DR

This paper introduces a predictable, high-performance hardware architecture and compiler toolchain for real-time neural network inference on a custom RISC-V multicore vector processor, addressing predictability and resource limitations.

Contribution

It presents a novel hardware design with predictable cores and a compiler-based scheduling approach to ensure real-time performance for neural network inference.

Findings

Achieves predictable WCET estimates for neural network inference

Provides a hardware architecture with local scratchpad memories and static memory scheduling

Demonstrates improved predictability and performance balance in real-time systems

Abstract

Neural networks are increasingly used in real-time systems, such as automated driving applications. This requires high-performance hardware with predictable timing behavior. State-of-the-art real-time hardware is limited in memory and compute resources. On the other hand, modern accelerator systems lack the necessary predictability properties, mainly due to interference in the memory subsystem. We present a new hardware architecture with an accompanying compiler-based deployment toolchain to close this gap between performance and predictability. The hardware architecture consists of a multicore vector processor with predictable cores, each with local scratchpad memories. A central management core facilitates access to shared external memory through a static schedule calculated at compile-time. The presented compiler exploits the fixed data flow of neural networks and WCET estimates of subtasks running on individual cores to compute this schedule. Through this approach, the WCET estimate of the overall system can be obtained from the subtask WCET estimates, data transfer times, and access times of the shared memory in conjunction with the schedule calculated by the compiler.