Real-Time In-Cabin Driver Behavior Recognition on Low-Cost Edge Hardware

TL;DR

This paper presents a real-time driver behavior recognition system optimized for low-cost edge hardware, combining a compact vision model, confounder-aware taxonomy, and temporal decision-making to enable effective in-cabin monitoring.

Contribution

It introduces a novel in-cabin driver behavior recognition pipeline optimized for Raspberry Pi 5 and Google Coral Edge TPU, achieving real-time performance with 17 behavior classes.

Findings

16 FPS on Raspberry Pi 5 with INT8 inference

25 FPS on Coral Edge TPU with ~40 ms latency

Validated in live in-vehicle tests

Abstract

In-cabin driver monitoring systems (DMS) must recognize distraction- and drowsiness-related behaviors with low latency under strict constraints on compute, power, and cost. We present a single-camera in-cabin driver behavior recognition system designed for deployment on two low-cost edge platforms: Raspberry Pi 5 (CPU-only) and the Google Coral development board with an Edge Tensor Processing Unit (Edge TPU) accelerator. The proposed pipeline combines (i) a compact per-frame vision model, (ii) a confounder-aware label taxonomy to reduce confusions among visually similar behaviors, and (iii) a temporal decision head that triggers alerts only when predictions are both confident and sustained. The system supports 17 behavior classes. Training and evaluation use licensed datasets plus in-house collection (over 800,000 labeled frames) with driver-disjoint splits, and we further validate the deployed system in live in-vehicle tests. End-to-end performance reaches approximately 16 FPS on Raspberry Pi 5 using 8-bit integer (INT8) inference (per-frame latency <60 ms) and approximately 25 FPS on Coral Edge TPU (end-to-end latency ~40 ms), enabling real-time monitoring and stable alert generation on embedded hardware. Finally, we discuss how reliable in-cabin perception can serve as an upstream signal for human-centered vehicle intelligence, including emerging agentic vehicle concepts.