Inside madupite: Technical Design and Performance

TL;DR

Madupite is a novel high-performance solver for large-scale discounted Markov decision processes, capable of exact solutions even beyond typical memory limits by leveraging distributed computing and customizable algorithms.

Contribution

Introduces madupite, a scalable, distributed solver for large MDPs that surpasses existing methods in efficiency, flexibility, and ability to handle problems exceeding standard memory capacities.

Findings

Madupite efficiently solves large-scale MDPs in distributed environments.

It outperforms existing solvers in scalability and speed.

It can handle problems exceeding laptop memory in near-undiscounted settings.

Abstract

In this work, we introduce and benchmark madupite, a newly proposed high-performance solver designed for large-scale discounted infinite-horizon Markov decision processes with finite state and action spaces. After a brief overview of the class of mathematical optimization methods on which madupite relies, we provide details on implementation choices, technical design and deployment. We then demonstrate its scalability and efficiency by showcasing its performance on the solution of Markov decision processes arising from different application areas, including epidemiology and classical control. Madupite sets a new standard as, to the best of our knowledge, it is the only solver capable of efficiently computing exact solutions for large-scale Markov decision processes, even when these exceed the memory capacity of modern laptops and operate in near-undiscounted settings. This is possible as madupite can work in a fully distributed manner and therefore leverage the memory storage and computation capabilities of modern high-performance computing clusters. This key feature enables the solver to efficiently handle problems of medium to large size in an exact manner instead of necessarily resorting to function approximations. Moreover, madupite is unique in allowing users to customize the solution algorithm to better exploit the specific structure of their problem, significantly accelerating convergence especially in large-discount factor settings. Overall, madupite represents a significant advancement, offering unmatched scalability and flexibility in solving large-scale Markov decision processes.