Recurrent Sum-Product-Max Networks for Decision Making in Perfectly-Observed Environments

TL;DR

This paper introduces recurrent sum-product-max networks (RSPMNs), a novel model that extends SPMNs for sequential decision-making, enabling data-driven, scalable, and effective policy learning in perfectly-observed environments.

Contribution

The paper proposes RSPMNs, a new recurrent architecture for SPMNs, including a structure learning algorithm and conditions for validity, tailored for sequential decision-making tasks.

Findings

RSPMNs generate near-optimal MEUs and policies in test domains.

They outperform recent batch-constrained reinforcement learning methods.

RSPMNs are scalable and suitable for offline decision-making in perfectly-observed environments.

Abstract

Recent investigations into sum-product-max networks (SPMN) that generalize sum-product networks (SPN) offer a data-driven alternative for decision making, which has predominantly relied on handcrafted models. SPMNs computationally represent a probabilistic decision-making problem whose solution scales linearly in the size of the network. However, SPMNs are not well suited for sequential decision making over multiple time steps. In this paper, we present recurrent SPMNs (RSPMN) that learn from and model decision-making data over time. RSPMNs utilize a template network that is unfolded as needed depending on the length of the data sequence. This is significant as RSPMNs not only inherit the benefits of SPMNs in being data driven and mostly tractable, they are also well suited for sequential problems. We establish conditions on the template network, which guarantee that the resulting SPMN is valid, and present a structure learning algorithm to learn a sound template network. We demonstrate that the RSPMNs learned on a testbed of sequential decision-making data sets generate MEUs and policies that are close to the optimal on perfectly-observed domains. They easily improve on a recent batch-constrained reinforcement learning method, which is important because RSPMNs offer a new model-based approach to offline reinforcement learning.