Control of Microrobots with Reinforcement Learning under On-Device Compute Constraints

TL;DR

This paper demonstrates reinforcement learning-based control of a microrobot on ultra-low-power hardware, optimizing gait and inference methods to operate within strict compute and power constraints, and validates on real terrain.

Contribution

It introduces a resource-aware RL control framework for microrobots, integrating domain randomization, quantization, and gait selection to enable on-device autonomous locomotion.

Findings

Reinforcement learning can effectively control microrobots within hardware constraints.

Quantization improves inference speed on low-power microcontrollers.

Gait mode selection based on power budget optimizes robot performance.

Abstract

An important function of autonomous microrobots is the ability to perform robust movement over terrain. This paper explores an edge ML approach to microrobot locomotion, allowing for on-device, lower latency control under compute, memory, and power constraints. This paper explores the locomotion of a sub-centimeter quadrupedal microrobot via reinforcement learning (RL) and deploys the resulting controller on an ultra-small system-on-chip (SoC), SC$\mu$M-3C, featuring an ARM Cortex-M0 microcontroller running at 5 MHz. We train a compact FP32 multilayer perceptron (MLP) policy with two hidden layers ($[128, 64]$) in a massively parallel GPU simulation and enhance robustness by utilizing domain randomization over simulation parameters. We then study integer (Int8) quantization (per-tensor and per-feature) to allow for higher inference update rates on our resource-limited hardware, and we connect hardware power budgets to achievable update frequency via a cycles-per-update model for inference on our Cortex-M0. We propose a resource-aware gait scheduling viewpoint: given a device power budget, we can select the gait mode (trot/intermediate/gallop) that maximizes expected RL reward at a corresponding feasible update frequency. Finally, we deploy our MLP policy on a real-world large-scale robot on uneven terrain, qualitatively noting that domain-randomized training can improve out-of-distribution stability. We do not claim real-world large-robot empirical zero-shot transfer in this work.