A Novel Hybrid Heuristic-Reinforcement Learning Optimization Approach for a Class of Railcar Shunting Problems

TL;DR

This paper introduces a hybrid heuristic-reinforcement learning framework to optimize complex railcar shunting problems involving multiple outbound trains and track access types, demonstrating improved efficiency and solution quality.

Contribution

It proposes a novel HHRL framework combining heuristics with Q-learning for railcar shunting, addressing a challenging combinatorial problem with two-sided track access and multiple locomotives.

Findings

HHRL outperforms traditional methods in solution quality.

The approach effectively reduces the state-action space.

Numerical experiments confirm the method's efficiency.

Abstract

Railcar shunting is a core planning task in freight railyards, where yard planners need to disassemble and reassemble groups of railcars to form outbound trains. Classification tracks with access from one side only can be considered as stack structures, where railcars are added and removed from only one end, leading to a last-in-first-out (LIFO) retrieval order. In contrast, two-sided tracks function like queue structures, allowing railcars to be added from one end and removed from the opposite end, following a first-in-first-out (FIFO) order. We consider a problem requiring assembly of multiple outbound trains using two locomotives in a railyard with two-sided classification track access. To address this combinatorially challenging problem class, we decompose the problem into two subproblems, each with one-sided classification track access and a locomotive on each side. We present a novel Hybrid Heuristic-Reinforcement Learning (HHRL) framework that integrates railway-specific heuristic solution approaches with a reinforcement learning method, specifically Q-learning. The proposed framework leverages methods to decrease the state-action space and guide exploration during reinforcement learning. The results of a series of numerical experiments demonstrate the efficiency and quality of the HHRL algorithm in both one-sided access, single-locomotive problems and two-sided access, two-locomotive problems.