OptimizedDP: An Efficient, User-friendly Library For Optimal Control and Dynamic Programming

TL;DR

OptimizedDP is a Python library that efficiently solves high-dimensional Hamilton-Jacobi PDEs and MDPs, enabling control and robotics applications previously considered computationally infeasible.

Contribution

The paper presents a high-performance, user-friendly Python library that significantly improves the efficiency and scalability of grid-based dynamic programming methods.

Findings

Achieves an order of magnitude faster execution times compared to similar tools.

Enables solving higher-dimensional PDEs and MDPs that were previously intractable.

Provides a flexible interface for various control and optimization problems.

Abstract

This paper introduces OptimizedDP, a high-performance software library for several common grid-based dynamic programming (DP) algorithms used in control theory and robotics. Specifically, OptimizedDP provides functions to numerically solve a class of time-dependent (dynamic) Hamilton-Jacobi (HJ) partial differential equations (PDEs), time-independent (static) HJ PDEs, and additionally value iteration for continuous action-state space Markov Decision Processes (MDP). The computational complexity of grid-based DP is exponential with respect to the number of grid or state space dimensions, and thus can have bad execution runtimes and memory usage whenapplied to large state spaces. We leverage the user-friendliness of Python for different problem specifications without sacrificing the efficiency of the core computation. This is achieved by implementing the core part of the code which the user does not see in heterocl, a framework we use to abstract away details of how computation is parallelized. Compared to similar toolboxes for level set methods that are used to solve the HJ PDE, our toolbox makes solving the PDE at higher dimensions possible as well as achieving an order of magnitude improvements in execution times, while keeping the interface easy for specifying different problem descriptions. Because of that, the toolbox has been adopted to solve control and optimization problems that were considered intractable before. Our toolbox is available publicly at https://github.com/SFU-MARS/optimized_dp.