Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment

TL;DR

This study empirically compares different time-step sizes in reinforcement learning for sepsis treatment, revealing that smaller steps generally improve policy performance and stability, challenging the conventional 4-hour standard.

Contribution

The paper systematically evaluates the impact of various time-step sizes on RL performance in sepsis management, introducing action re-mapping for fair comparison across different temporal granularities.

Findings

Finer time-steps ($$h, 2h) yield better policy performance.

Performance trends vary with learning setups and time-step sizes.

Smaller time-steps improve stability and policy quality.

Abstract

Existing studies on reinforcement learning (RL) for sepsis management have mostly followed an established problem setup, in which patient data are aggregated into 4-hour time steps. Although concerns have been raised regarding the coarseness of this time-step size, which might distort patient dynamics and lead to suboptimal treatment policies, the extent to which this is a problem in practice remains unexplored. In this work, we conducted empirical experiments for a controlled comparison of four time-step sizes ($\Delta t\!=\!1,2,4,8$ h) on this domain, following an identical offline RL pipeline. To enable a fair comparison across time-step sizes, we designed action re-mapping methods that allow for evaluation of policies on datasets with different time-step sizes, and conducted cross-$\Delta t$ model selections under two policy learning setups. Our goal was to quantify how time-step size influences state representation learning, behavior cloning, policy training, and off-policy evaluation. Our results show that performance trends across $\Delta t$ vary as learning setups change, while policies learned at finer time-step sizes ($\Delta t = 1$ h and $2$ h) using a static behavior policy achieve the overall best performance and stability. Our work highlights time-step size as a core design choice in offline RL for healthcare and provides evidence supporting alternatives beyond the conventional 4-hour setup.