Dynamic Observation Policies in Observation Cost-Sensitive Reinforcement Learning

TL;DR

This paper introduces a novel reinforcement learning approach that reduces measurement costs by dynamically deciding when to observe the environment, leading to more efficient policies in complex tasks.

Contribution

It proposes the Deep Dynamic Multi-Step Observationless Agent (DMSOA), a new method that minimizes observation costs while maintaining or improving policy performance.

Findings

DMSOA outperforms existing methods in OpenAI gym and Atari Pong environments.

DMSOA learns policies with fewer decision steps and measurements.

The approach reduces observation costs without sacrificing task success.

Abstract

Reinforcement learning (RL) has been shown to learn sophisticated control policies for complex tasks including games, robotics, heating and cooling systems and text generation. The action-perception cycle in RL, however, generally assumes that a measurement of the state of the environment is available at each time step without a cost. In applications such as materials design, deep-sea and planetary robot exploration and medicine, however, there can be a high cost associated with measuring, or even approximating, the state of the environment. In this paper, we survey the recently growing literature that adopts the perspective that an RL agent might not need, or even want, a costly measurement at each time step. Within this context, we propose the Deep Dynamic Multi-Step Observationless Agent (DMSOA), contrast it with the literature and empirically evaluate it on OpenAI gym and Atari Pong environments. Our results, show that DMSOA learns a better policy with fewer decision steps and measurements than the considered alternative from the literature.