Circuits and Systems for Embodied AI: Exploring uJ Multi-Modal Perception for Nano-UAVs on the Kraken Shield

TL;DR

This paper presents a novel multi-modal perception system for nano-UAVs using the Kraken shield, enabling real-time, energy-efficient AI inference for navigation and object detection on ultra-light devices.

Contribution

It introduces a new multi-sensor board with specialized neural network accelerators for nano-UAVs, demonstrating real-time multi-modal perception capabilities within strict size and energy constraints.

Findings

Kraken can perform depth estimation at 1.02k inf/s with 18 μJ per inference.

Object classification runs at 10k inf/s with 6 μJ per inference.

Obstacle avoidance inference at 221 frames per second with 750 μJ per frame.

Abstract

Embodied artificial intelligence (AI) requires pushing complex multi-modal models to the extreme edge for time-constrained tasks such as autonomous navigation of robots and vehicles. On small form-factor devices, e.g., nano-sized unmanned aerial vehicles (UAVs), such challenges are exacerbated by stringent constraints on energy efficiency and weight. In this paper, we explore embodied multi-modal AI-based perception for Nano-UAVs with the Kraken shield, a 7g multi-sensor (frame-based and event-based imagers) board based on Kraken, a 22 nm SoC featuring multiple acceleration engines for multi-modal event and frame-based inference based on spiking (SNN) and ternary (TNN) neural networks, respectively. Kraken can execute SNN real-time inference for depth estimation at 1.02k inf/s, 18 {\mu}J/inf, TNN real-time inference for object classification at 10k inf/s, 6 {\mu}J/inf, and real-time inference for obstacle avoidance at 221 frame/s, 750 {\mu}J/inf.