A Scheme for Dynamic Risk-Sensitive Sequential Decision Making

TL;DR

This paper introduces a neural network-based scheme for risk-sensitive sequential decision making in dynamic environments, capable of handling constraints and varying parameters by estimating risk through return variance.

Contribution

It proposes a novel approach combining neural approximation and state-augmentation to address risk-sensitive decision making with dynamic parameters and constraints.

Findings

Most risk measures can be estimated using return variance.

State-augmentation enables risk-sensitive solutions for MDPs with stochastic rewards.

Numerical experiments validate the effectiveness of the proposed scheme.

Abstract

We present a scheme for sequential decision making with a risk-sensitive objective and constraints in a dynamic environment. A neural network is trained as an approximator of the mapping from parameter space to space of risk and policy with risk-sensitive constraints. For a given risk-sensitive problem, in which the objective and constraints are, or can be estimated by, functions of the mean and variance of return, we generate a synthetic dataset as training data. Parameters defining a targeted process might be dynamic, i.e., they might vary over time, so we sample them within specified intervals to deal with these dynamics. We show that: i). Most risk measures can be estimated using return variance; ii). By virtue of the state-augmentation transformation, practical problems modeled by Markov decision processes with stochastic rewards can be solved in a risk-sensitive scenario; and iii). The proposed scheme is validated by a numerical experiment.