Dynamic Resource Management for Providing QoS in Drone Delivery Systems

TL;DR

This paper presents a deep reinforcement learning approach for dynamic drone-to-PDC allocation in drone delivery systems, ensuring QoS guarantees and efficient UAV utilization amid variable demand patterns.

Contribution

It introduces a queueing theoretic model combined with deep reinforcement learning to optimize UAV re-allocation with probabilistic QoS guarantees in drone delivery.

Findings

Outperforms baseline algorithms in diverse demand scenarios.

Maintains QoS constraints across all tested demand patterns.

Reduces average UAV usage while meeting service quality.

Abstract

Drones have been considered as an alternative means of package delivery to reduce the delivery cost and time. Due to the battery limitations, the drones are best suited for last-mile delivery, i.e., the delivery from the package distribution centers (PDCs) to the customers. Since a typical delivery system consists of multiple PDCs, each having random and time-varying demands, the dynamic drone-to-PDC allocation would be of great importance in meeting the demand in an efficient manner. In this paper, we study the dynamic UAV assignment problem for a drone delivery system with the goal of providing measurable Quality of Service (QoS) guarantees. We adopt a queueing theoretic approach to model the customer-service nature of the problem. Furthermore, we take a deep reinforcement learning approach to obtain a dynamic policy for the re-allocation of the UAVs. This policy guarantees a probabilistic upper-bound on the queue length of the packages waiting in each PDC, which is beneficial from both the service provider's and the customers' viewpoints. We evaluate the performance of our proposed algorithm by considering three broad arrival classes, including Bernoulli, Time-Varying Bernoulli, and Markov-Modulated Bernoulli arrivals. Our results show that the proposed method outperforms the baselines, particularly in scenarios with Time-Varying and Markov-Modulated Bernoulli arrivals, which are more representative of real-world demand patterns. Moreover, our algorithm satisfies the QoS constraints in all the studied scenarios while minimizing the average number of UAVs in use.