Split CNN Inference on Networked Microcontrollers

TL;DR

This paper introduces a split inference system for CNNs on networked microcontrollers, enabling collaborative execution across multiple devices to overcome memory constraints while maintaining latency.

Contribution

It proposes a novel sub-layer splitting approach for CNN inference on MCUs, distributing model parameters and activations across devices to reduce peak RAM usage.

Findings

Enables CNN inference on multiple MCUs previously infeasible on a single device.

Reduces peak RAM usage per MCU while maintaining inference latency.

Successfully tested with MobileNetV2 on up to 8 MCUs.

Abstract

Running deep neural networks on microcontroller units (MCUs) is severely constrained by limited memory resources. While TinyML techniques reduce model size and computation, they often fail in practice due to excessive peak Random Access Memory (RAM) usage during inference, dominated by intermediate activations. As a result, many models remain infeasible on standalone MCUs. In this work, we present a fine-grained split inference system for networked MCUs that enables collaborative inference of Convolutional Neural Networks (CNN) models across multiple devices. Our key insight is that breaking the memory bottleneck requires splitting inference at sub-layer granularity rather than at layer boundaries. We reinterpret pre-trained models to enable kernel-wise and neuron-wise partitioning, and distribute both model parameters and intermediate activations across multiple MCUs. A lightweight, resource-aware coordinator orchestrates the inference across MCU devices with heterogeneous resources. We implement the proposed system on a real testbed and evaluate it on up to 8 MCUs using MobileNetV2, a representative CNN model. Our experimental results show that CNN models infeasible on a single MCU can be executed across networked MCUs, reducing the per-MCU peak RAM usage while maintaining the practical end-to-end inference latency. All the source code of this work can be found here: https://github.com/shashsuresh/split-inference-on-MCUs.