Multi-Agent Reinforcement Learning Scheduling to Support Low Latency in Teleoperated Driving

TL;DR

This paper introduces multi-agent reinforcement learning algorithms to optimize radio resource allocation, significantly reducing end-to-end latency in teleoperated driving scenarios with multiple vehicles.

Contribution

It proposes novel MARL scheduling algorithms based on PPO for dynamic radio resource management in teleoperated driving, comparing decentralized and centralized training paradigms.

Findings

MAPPO with greedy allocation achieves lowest latency.

Centralized training outperforms decentralized in high vehicle density.

MARL approaches adapt effectively to network conditions.

Abstract

The teleoperated driving (TD) scenario comes with stringent Quality of Service (QoS) communication constraints, especially in terms of end-to-end (E2E) latency and reliability. In this context, Predictive Quality of Service (PQoS), possibly combined with Reinforcement Learning (RL) techniques, is a powerful tool to estimate QoS degradation and react accordingly. For example, an intelligent agent can be trained to select the optimal compression configuration for automotive data, and reduce the file size whenever QoS conditions deteriorate. However, compression may inevitably compromise data quality, with negative implications for the TD application. An alternative strategy involves operating at the Radio Access Network (RAN) level to optimize radio parameters based on current network conditions, while preserving data quality. In this paper, we propose Multi-Agent Reinforcement Learning (MARL) scheduling algorithms, based on Proximal Policy Optimization (PPO), to dynamically and intelligently allocate radio resources to minimize E2E latency in a TD scenario. We evaluate two training paradigms, i.e., decentralized learning with local observations (IPPO) vs. centralized aggregation (MAPPO), in conjunction with two resource allocation strategies, i.e., proportional allocation (PA) and greedy allocation (GA). We prove via ns-3 simulations that MAPPO, combined with GA, achieves the best results in terms of latency, especially as the number of vehicles increases.