DRL4Route: A Deep Reinforcement Learning Framework for Pick-up and Delivery Route Prediction

TL;DR

This paper introduces DRL4Route, a reinforcement learning framework for pick-up and delivery route prediction, which improves accuracy by aligning training with practical test criteria, outperforming existing deep learning models.

Contribution

It is the first to apply reinforcement learning to route prediction, integrating non-differentiable objective optimization with deep learning for better practical performance.

Findings

Improves Location Square Deviation (LSD) by 0.9%-2.7%.

Enhances Accuracy@3 (ACC@3) by 2.4%-3.2%.

Demonstrates effectiveness through offline and online experiments.

Abstract

Pick-up and Delivery Route Prediction (PDRP), which aims to estimate the future service route of a worker given his current task pool, has received rising attention in recent years. Deep neural networks based on supervised learning have emerged as the dominant model for the task because of their powerful ability to capture workers' behavior patterns from massive historical data. Though promising, they fail to introduce the non-differentiable test criteria into the training process, leading to a mismatch in training and test criteria. Which considerably trims down their performance when applied in practical systems. To tackle the above issue, we present the first attempt to generalize Reinforcement Learning (RL) to the route prediction task, leading to a novel RL-based framework called DRL4Route. It combines the behavior-learning abilities of previous deep learning models with the non-differentiable objective optimization ability of reinforcement learning. DRL4Route can serve as a plug-and-play component to boost the existing deep learning models. Based on the framework, we further implement a model named DRL4Route-GAE for PDRP in logistic service. It follows the actor-critic architecture which is equipped with a Generalized Advantage Estimator that can balance the bias and variance of the policy gradient estimates, thus achieving a more optimal policy. Extensive offline experiments and the online deployment show that DRL4Route-GAE improves Location Square Deviation (LSD) by 0.9%-2.7%, and Accuracy@3 (ACC@3) by 2.4%-3.2% over existing methods on the real-world dataset.