Active Measure Reinforcement Learning for Observation Cost Minimization

TL;DR

This paper introduces Active Measure Reinforcement Learning (Amrl), enabling agents to balance observation costs and rewards by learning to minimize observation costs while maximizing returns.

Contribution

The paper presents the Amrl framework, allowing RL agents to learn policies that optimize the trade-off between observation costs and rewards during training.

Findings

Amrl-Q agents learn policies and state estimators in parallel.

Agents shift from costly measurements to estimators during training.

Amrl-Q achieves higher costed return than standard methods.

Abstract

Standard reinforcement learning (RL) algorithms assume that the observation of the next state comes instantaneously and at no cost. In a wide variety of sequential decision making tasks ranging from medical treatment to scientific discovery, however, multiple classes of state observations are possible, each of which has an associated cost. We propose the active measure RL framework (Amrl) as an initial solution to this problem where the agent learns to maximize the costed return, which we define as the discounted sum of rewards minus the sum of observation costs. Our empirical evaluation demonstrates that Amrl-Q agents are able to learn a policy and state estimator in parallel during online training. During training the agent naturally shifts from its reliance on costly measurements of the environment to its state estimator in order to increase its reward. It does this without harm to the learned policy. Our results show that the Amrl-Q agent learns at a rate similar to standard Q-learning and Dyna-Q. Critically, by utilizing an active strategy, Amrl-Q achieves a higher costed return.