RLOR: A Flexible Framework of Deep Reinforcement Learning for Operation Research

TL;DR

RLOR is a flexible deep reinforcement learning framework tailored for operation research, enabling easier integration of recent RL advances and customizable architectures, demonstrated through vehicle routing problem solutions with significant training speed improvements.

Contribution

The paper introduces RLOR, a versatile framework that enhances deep reinforcement learning applications in operation research by allowing flexible model architecture customization and incorporating recent RL techniques.

Findings

Re-implemented Attention Model with PPO, achieving 8x faster training.

Demonstrated the framework's flexibility in adapting RL models for operation research.

Publicly released code for reproducibility and further research.

Abstract

Reinforcement learning has been applied in operation research and has shown promise in solving large combinatorial optimization problems. However, existing works focus on developing neural network architectures for certain problems. These works lack the flexibility to incorporate recent advances in reinforcement learning, as well as the flexibility of customizing model architectures for operation research problems. In this work, we analyze the end-to-end autoregressive models for vehicle routing problems and show that these models can benefit from the recent advances in reinforcement learning with a careful re-implementation of the model architecture. In particular, we re-implemented the Attention Model and trained it with Proximal Policy Optimization (PPO) in CleanRL, showing at least 8 times speed up in training time. We hereby introduce RLOR, a flexible framework for Deep Reinforcement Learning for Operation Research. We believe that a flexible framework is key to developing deep reinforcement learning models for operation research problems. The code of our work is publicly available at https://github.com/cpwan/RLOR.