Kraken: A Direct Event/Frame-Based Multi-sensor Fusion SoC for Ultra-Efficient Visual Processing in Nano-UAVs

TL;DR

Kraken is a low-power, multi-sensor SoC for nano-UAVs that enables complex visual processing tasks like object detection and navigation using event-based and frame-based sensors, all on a single chip.

Contribution

This paper introduces Kraken, a novel ultra-efficient multi-sensor SoC integrating event-based and frame-based sensors with specialized accelerators for nano-UAV visual tasks.

Findings

Supports sparse event-driven SNN inference at sub-uJ/inf energy levels.

Performs frame-based DNN inference with high efficiency using mixed-precision extensions.

Integrates multiple acceleration engines and peripherals for versatile sensor interfacing.

Abstract

Small-size unmanned aerial vehicles (UAV) have the potential to dramatically increase safety and reduce cost in applications like critical infrastructure maintenance and post-disaster search and rescue. Many scenarios require UAVs to shrink toward nano and pico-size form factors. The key open challenge to achieve true autonomy on Nano-UAVs is to run complex visual tasks like object detection, tracking, navigation and obstacle avoidance fully on board, at high speed and robustness, under tight payload and power constraints. With the Kraken SoC, fabricated in 22nm FDX technology, we demonstrate a multi-visual-sensor capability exploiting both event-based and BW/RGB imagers, combining their output for multi-functional visual tasks previously impossible on a single low-power chip for Nano-UAVs. Kraken is an ultra-low-power, heterogeneous SoC architecture integrating three acceleration engines and a vast set of peripherals to enable efficient interfacing with standard frame-based sensors and novel event-based DVS. Kraken enables highly sparse event-driven sub-uJ/inf SNN inference on a dedicated neuromorphic energy-proportional accelerator. Moreover, it can perform frame-based inference by combining a 1.8TOp\s\W 8-cores RISC-V processor cluster with mixed-precision DNN extensions with a 1036TOp\s\W} TNN accelerator.