Hidden Symmetries and Model Reduction in Markov Decision Processes: Explained and Applied to the Multi-period Newsvendor Problem

TL;DR

This paper introduces the concept of hidden symmetries in Markov decision processes, which, when revealed by modifying reward structures, enable more effective model reduction than traditional symmetry methods, demonstrated on the multi-period newsvendor problem.

Contribution

The paper defines hidden symmetries in MDPs, proposes a method to reveal them through reward modification, and applies this to improve model reduction in the multi-period newsvendor problem.

Findings

Hidden symmetries enable better model reduction than accessible symmetries.

Revealing hidden symmetries involves altering the reward structure.

Application to the multi-period newsvendor problem demonstrates effectiveness.

Abstract

Symmetry breaking is a common approach for model reduction of Markov decision processes (MDPs). This approach only uses directly accessible symmetries such as geometric symmetries. For some MDPs, it is possible to transform them equivalently such that symmetries become accessible -- we call this type of symmetries hidden symmetries. For these MDPs, hidden symmetries allow substantially better model reduction compared to directly accessible symmetries. The main idea is to reveal a hidden symmetry by altering the reward structure and then exploit the revealed symmetry by forming a quotient MDP. The quotient MDP is the reduced MDP, since it is sufficient to solve the quotient MDP instead of the original one. In this paper, we introduce hidden symmetries and the associated concept of model reduction. We demonstrate this concept on the multi-period newsvendor problem, which is the newsvendor problem considered over an infinite number of days. In this way, we show that hidden symmetries can reduce problems that directly accessible symmetries cannot, and present a basic idea of revealing hidden symmetries in multi-period problems. The presented approach can be extended to more sophisticated problems.