Model-aided Deep Reinforcement Learning for Sample-efficient UAV Trajectory Design in IoT Networks

TL;DR

This paper introduces a model-aided deep Q-learning method that significantly reduces training data requirements for UAV trajectory optimization in IoT networks, enabling efficient data collection without prior channel knowledge.

Contribution

It proposes a novel model-aided DRL approach using environment modeling and anchor nodes to cut down training data needs for UAV trajectory design.

Findings

Requires at least ten times fewer training samples than standard DRL methods.

Achieves comparable data collection performance with limited prior knowledge.

Demonstrates practical viability for real-world UAV IoT applications.

Abstract

Deep Reinforcement Learning (DRL) is gaining attention as a potential approach to design trajectories for autonomous unmanned aerial vehicles (UAV) used as flying access points in the context of cellular or Internet of Things (IoT) connectivity. DRL solutions offer the advantage of on-the-go learning hence relying on very little prior contextual information. A corresponding drawback however lies in the need for many learning episodes which severely restricts the applicability of such approach in real-world time- and energy-constrained missions. Here, we propose a model-aided deep Q-learning approach that, in contrast to previous work, considerably reduces the need for extensive training data samples, while still achieving the overarching goal of DRL, i.e to guide a battery-limited UAV on an efficient data harvesting trajectory, without prior knowledge of wireless channel characteristics and limited knowledge of wireless node locations. The key idea consists in using a small subset of nodes as anchors (i.e. with known location) and learning a model of the propagation environment while implicitly estimating the positions of regular nodes. Interaction with the model allows us to train a deep Q-network (DQN) to approximate the optimal UAV control policy. We show that in comparison with standard DRL approaches, the proposed model-aided approach requires at least one order of magnitude less training data samples to reach identical data collection performance, hence offering a first step towards making DRL a viable solution to the problem.