Speeding up Policy Simulation in Supply Chain RL

TL;DR

This paper introduces Picard Iteration, an iterative, GPU-accelerated algorithm that significantly speeds up policy simulation in supply chain RL by enabling batched evaluations and rapid convergence.

Contribution

The paper presents a novel iterative algorithm for policy simulation that leverages parallel processing and caching, achieving large speedups in supply chain RL applications.

Findings

Achieves 400x speedup on large-scale supply chain problems

Converges in a small number of iterations regardless of horizon length

Effective in other RL environments beyond supply chains

Abstract

Simulating a single trajectory of a dynamical system under some state-dependent policy is a core bottleneck in policy optimization (PO) algorithms. The many inherently serial policy evaluations that must be performed in a single simulation constitute the bulk of this bottleneck. In applying PO to supply chain optimization (SCO) problems, simulating a single sample path corresponding to one month of a supply chain can take several hours. We present an iterative algorithm to accelerate policy simulation, dubbed Picard Iteration. This scheme carefully assigns policy evaluation tasks to independent processes. Within an iteration, any given process evaluates the policy only on its assigned tasks while assuming a certain "cached" evaluation for other tasks; the cache is updated at the end of the iteration. Implemented on GPUs, this scheme admits batched evaluation of the policy across a single trajectory. We prove that the structure afforded by many SCO problems allows convergence in a small number of iterations independent of the horizon. We demonstrate practical speedups of 400x on large-scale SCO problems even with a single GPU, and also demonstrate practical efficacy in other RL environments.