madupite: A High-Performance Distributed Solver for Large-Scale Markov Decision Processes

TL;DR

madupite is a high-performance distributed solver for large-scale Markov Decision Processes, enabling efficient solutions for high-dimensional problems using modern computing resources and supporting diverse solution methods.

Contribution

It introduces madupite, a scalable distributed solver that overcomes limitations of existing tools by leveraging HPC resources and supporting multiple solution techniques.

Findings

Supports MDPs with millions of states

Efficiently distributes computation across multiple nodes

Offers a flexible Python API with a C++ core

Abstract

This paper introduces madupite, a high-performance distributed solver for large-scale Markov Decision Processes (MDPs). MDPs are widely used to model complex dynamical systems in various fields, including finance, epidemiology, and traffic control. However, real-world applications often result in extremely high-dimensional MDPs, leading to the curse of dimensionality, which is typically addressed through function approximators like neural networks. While existing solvers such as pymdptoolbox and mdpsolver provide tools for solving MDPs, they either lack scalability, support for distributed computing, or flexibility in solution methods. madupite is designed to overcome these limitations by leveraging modern high-performance computing resources. It efficiently distributes memory load and computation across multiple nodes, supports a diverse set of solution methods, and offers a user-friendly Python API while maintaining a C++ core for optimal performance. With the ability to solve MDPs with millions of states, madupite provides researchers and engineers with a powerful tool to tackle large-scale decision-making problems with greater efficiency and flexibility.