An End-to-End Deep Reinforcement Learning Approach for Solving the Traveling Salesman Problem with Drones

TL;DR

This paper introduces a hierarchical deep reinforcement learning framework with a Transformer-inspired encoder and minimal gated decoder to efficiently solve the complex TSP with Drones, outperforming existing methods in solution quality and training efficiency.

Contribution

It presents a novel hierarchical Actor-Critic deep reinforcement learning model with a specialized attention mechanism for the TSP-D problem, improving solution quality and training speed.

Findings

Achieves competitive or superior solutions on TSP-D instances of various scales.

Reduces training time significantly compared to existing reinforcement learning algorithms.

Provides faster computation times than traditional heuristic algorithms.

Abstract

The emergence of truck-drone collaborative systems in last-mile logistics has positioned the Traveling Salesman Problem with Drones (TSP-D) as a pivotal extension of classical routing optimization, where synchronized vehicle coordination promises substantial operational efficiency and reduced environmental impact, yet introduces NP-hard combinatorial complexity beyond the reach of conventional optimization paradigms. Deep reinforcement learning offers a theoretically grounded framework to address TSP-D's inherent challenges through self-supervised policy learning and adaptive decision-making. This study proposes a hierarchical Actor-Critic deep reinforcement learning framework for solving the TSP-D problem. The architecture consists of two primary components: a Transformer-inspired encoder and an efficient Minimal Gated Unit decoder. The encoder incorporates a novel, optimized k-nearest neighbors sparse attention mechanism specifically for focusing on relevant spatial relationships, further enhanced by the integration of global node features. The Minimal Gated Unit decoder processes these encoded representations to efficiently generate solution sequences. The entire framework operates within an asynchronous advantage actor-critic paradigm. Experimental results show that, on benchmark TSP-D instances of various scales (N=10 to 100), the proposed model can obtain competitive or even superior solutions in shorter average computation times compared to high-performance heuristic algorithms and existing reinforcement learning methods. Moreover, compared to advanced reinforcement learning algorithm benchmarks, the proposed framework significantly reduces the total training time required while achieving superior final performance, highlighting its notable advantage in training efficiency.