Lotus: learning-based online thermal and latency variation management for two-stage detectors on edge devices

TL;DR

Lotus is an online learning-based framework that dynamically manages thermal and latency variations in two-stage detectors on edge devices using deep reinforcement learning, improving inference stability and hardware efficiency.

Contribution

The paper introduces Lotus, a novel DRL-based framework that jointly scales CPU and GPU frequencies to stabilize latency and reduce thermal issues in two-stage detectors on edge hardware.

Findings

Significantly reduces latency variation across different settings.

Achieves faster inference speeds while maintaining lower temperatures.

Demonstrates effectiveness on NVIDIA Jetson and mobile platforms.

Abstract

Two-stage object detectors exhibit high accuracy and precise localization, especially for identifying small objects that are favorable for various edge applications. However, the high computation costs associated with two-stage detection methods cause more severe thermal issues on edge devices, incurring dynamic runtime frequency change and thus large inference latency variations. Furthermore, the dynamic number of proposals in different frames leads to various computations over time, resulting in further latency variations. The significant latency variations of detectors on edge devices can harm user experience and waste hardware resources. To avoid thermal throttling and provide stable inference speed, we propose Lotus, a novel framework that is tailored for two-stage detectors to dynamically scale CPU and GPU frequencies jointly in an online manner based on deep reinforcement learning (DRL). To demonstrate the effectiveness of Lotus, we implement it on NVIDIA Jetson Orin Nano and Mi 11 Lite mobile platforms. The results indicate that Lotus can consistently and significantly reduce latency variation, achieve faster inference, and maintain lower CPU and GPU temperatures under various settings.