POMDP inference and robust solution via deep reinforcement learning: An application to railway optimal maintenance

TL;DR

This paper introduces a combined inference and deep reinforcement learning framework for solving POMDPs with uncertain models, demonstrated on railway maintenance planning, enhancing robustness and applicability in real-world scenarios.

Contribution

It presents a novel approach integrating Bayesian inference with deep RL for POMDPs, including a hybrid model-based/model-free method and application to railway maintenance.

Findings

The framework effectively infers POMDP parameters from data.

Deep RL solutions are robust to model uncertainty via domain randomization.

Hybrid approaches outperform purely model-free methods in the application.

Abstract

Partially Observable Markov Decision Processes (POMDPs) can model complex sequential decision-making problems under stochastic and uncertain environments. A main reason hindering their broad adoption in real-world applications is the lack of availability of a suitable POMDP model or a simulator thereof. Available solution algorithms, such as Reinforcement Learning (RL), require the knowledge of the transition dynamics and the observation generating process, which are often unknown and non-trivial to infer. In this work, we propose a combined framework for inference and robust solution of POMDPs via deep RL. First, all transition and observation model parameters are jointly inferred via Markov Chain Monte Carlo sampling of a hidden Markov model, which is conditioned on actions, in order to recover full posterior distributions from the available data. The POMDP with uncertain parameters is then solved via deep RL techniques with the parameter distributions incorporated into the solution via domain randomization, in order to develop solutions that are robust to model uncertainty. As a further contribution, we compare the use of transformers and long short-term memory networks, which constitute model-free RL solutions, with a model-based/model-free hybrid approach. We apply these methods to the real-world problem of optimal maintenance planning for railway assets.