Comparative Analysis of Discrete and Continuous Action Spaces in Reservoir Management and Inventory Control Problems

TL;DR

This paper compares discrete and continuous action spaces in reservoir management and inventory control, analyzing computational trade-offs, and introduces a symbolic method for solving hybrid MDPs with new data structures.

Contribution

It provides a detailed comparison of action space discretization effects and introduces a novel symbolic approach using XADD for hybrid MDPs.

Findings

Finer discretizations approach continuous performance but increase computational costs.

Exponential growth in complexity with more discrete actions and inventory items.

The XADD data structure effectively manages piecewise symbolic value functions.

Abstract

This paper presents a comparative analysis of discrete and continuous action spaces within the contexts of reservoir management and inventory control problems. We explore the computational trade-offs between discrete action discretizations and continuous action settings, focusing on their effects on time complexity and space requirements across different horizons. Our analysis includes a detailed evaluation of discretization levels in reservoir management, highlighting that finer discretizations approach the performance of continuous actions but at increased computational costs. For inventory control, we investigate deterministic and stochastic demand scenarios, demonstrating the exponential growth in time and space with increasing discrete actions and inventory items. We also introduce a novel symbolic approach for solving continuous problems in hybrid MDPs (H-MDPs), utilizing a new XADD data structure to manage piecewise symbolic value functions. Our results underscore the challenges of scaling solutions and provide insights into efficient handling of discrete and continuous action spaces in complex decision problems. Future research directions include exploring heuristic search methods and improved approximations for enhancing the practicality of exact solutions.