Towards Cost Sensitive Decision Making

TL;DR

This paper introduces Active-Acquisition POMDPs, enabling RL agents to actively select features to acquire, balancing costs and rewards, and employs a deep generative model for feature imputation to improve decision-making.

Contribution

It proposes a novel Active-Acquisition POMDP framework and hierarchical RL algorithms utilizing deep generative models for feature imputation, advancing decision-making in uncertain environments.

Findings

Outperforms existing POMDP-RL methods in experiments

Effectively balances feature acquisition costs and decision rewards

Uses deep generative models to improve belief estimation

Abstract

Many real-world situations allow for the acquisition of additional relevant information when making decisions with limited or uncertain data. However, traditional RL approaches either require all features to be acquired beforehand (e.g. in a MDP) or regard part of them as missing data that cannot be acquired (e.g. in a POMDP). In this work, we consider RL models that may actively acquire features from the environment to improve the decision quality and certainty, while automatically balancing the cost of feature acquisition process and the reward of task decision process. We propose the Active-Acquisition POMDP and identify two types of the acquisition process for different application domains. In order to assist the agent in the actively-acquired partially-observed environment and alleviate the exploration-exploitation dilemma, we develop a model-based approach, where a deep generative model is utilized to capture the dependencies of the features and impute the unobserved features. The imputations essentially represent the beliefs of the agent. Equipped with the dynamics model, we develop hierarchical RL algorithms to resolve both types of the AA-POMDPs. Empirical results demonstrate that our approach achieves considerably better performance than existing POMDP-RL solutions.