R^3: On-device Real-Time Deep Reinforcement Learning for Autonomous Robotics

TL;DR

This paper introduces R^3, a comprehensive system for managing timing, memory, and performance trade-offs in on-device real-time deep reinforcement learning for autonomous robots, ensuring efficient adaptation in dynamic environments.

Contribution

R^3 is a holistic solution that dynamically balances timing, memory, and algorithm performance in on-device DRL training through feedback loops, memory management, and runtime coordination.

Findings

R^3 maintains consistent latency and timing predictability across hardware platforms.

It reduces memory footprint, enabling larger replay buffers and improved learning.

R^3 demonstrates effectiveness in autonomous vehicle simulation environments.

Abstract

Autonomous robotic systems, like autonomous vehicles and robotic search and rescue, require efficient on-device training for continuous adaptation of Deep Reinforcement Learning (DRL) models in dynamic environments. This research is fundamentally motivated by the need to understand and address the challenges of on-device real-time DRL, which involves balancing timing and algorithm performance under memory constraints, as exposed through our extensive empirical studies. This intricate balance requires co-optimizing two pivotal parameters of DRL training -- batch size and replay buffer size. Configuring these parameters significantly affects timing and algorithm performance, while both (unfortunately) require substantial memory allocation to achieve near-optimal performance. This paper presents R^3, a holistic solution for managing timing, memory, and algorithm performance in on-device real-time DRL training. R^3 employs (i) a deadline-driven feedback loop with dynamic batch sizing for optimizing timing, (ii) efficient memory management to reduce memory footprint and allow larger replay buffer sizes, and (iii) a runtime coordinator guided by heuristic analysis and a runtime profiler for dynamically adjusting memory resource reservations. These components collaboratively tackle the trade-offs in on-device DRL training, improving timing and algorithm performance while minimizing the risk of out-of-memory (OOM) errors. We implemented and evaluated R^3 extensively across various DRL frameworks and benchmarks on three hardware platforms commonly adopted by autonomous robotic systems. Additionally, we integrate R^3 with a popular realistic autonomous car simulator to demonstrate its real-world applicability. Evaluation results show that R^3 achieves efficacy across diverse platforms, ensuring consistent latency performance and timing predictability with minimal overhead.