Exploring Optimal Control With Observations at a Cost

TL;DR

This paper investigates optimal observation and intervention strategies in reinforcement learning, demonstrating that augmenting states with counters and using neural network-based predictions enhances performance and convergence in a simulated control task.

Contribution

It introduces methods to improve RL performance by modeling missing data more effectively, addressing observation costs and imputation ambiguity in healthcare-related RL tasks.

Findings

Augmenting state with counters improves predictive performance.

Neural network-based dynamics models outperform LOCF in state prediction.

Enhanced methods lead to faster convergence and lower variance.

Abstract

There has been a current trend in reinforcement learning for healthcare literature, where in order to prepare clinical datasets, researchers will carry forward the last results of the non-administered test known as the last-observation-carried-forward (LOCF) value to fill in gaps, assuming that it is still an accurate indicator of the patient's current state. These values are carried forward without maintaining information about exactly how these values were imputed, leading to ambiguity. Our approach models this problem using OpenAI Gym's Mountain Car and aims to address when to observe the patient's physiological state and partly how to intervene, as we have assumed we can only act after following an observation. So far, we have found that for a last-observation-carried-forward implementation of the state space, augmenting the state with counters for each state variable tracking the time since last observation was made, improves the predictive performance of an agent, supporting the notion of "informative missingness", and using a neural network based Dynamics Model to predict the most probable next state value of non-observed state variables instead of carrying forward the last observed value through LOCF further improves the agent's performance, leading to faster convergence and reduced variance.