Reinforcement Learning for Control of Non-Markovian Cellular Population Dynamics

TL;DR

This paper applies reinforcement learning to control cell populations with non-Markovian dynamics, enabling effective drug dosing strategies even with complex memory effects and uncertain system parameters.

Contribution

It introduces a model-free deep RL approach for controlling non-Markovian cellular dynamics, achieving near-optimal solutions in complex, uncertain environments.

Findings

RL recovers exact solutions in known models

RL controls populations under noise and memory variability

Robust RL strategies outperform traditional methods

Abstract

Many organisms and cell types, from bacteria to cancer cells, exhibit a remarkable ability to adapt to fluctuating environments. Additionally, cells can leverage a memory of past environments to better survive previously-encountered stressors. From a control perspective, this adaptability poses significant challenges in driving cell populations toward extinction, and thus poses an open question with great clinical significance. In this work, we focus on drug dosing in cell populations exhibiting phenotypic plasticity. For specific dynamical models switching between resistant and susceptible states, exact solutions are known. However, when the underlying system parameters are unknown, and for complex memory-based systems, obtaining the optimal solution is currently intractable. To address this challenge, we apply reinforcement learning (RL) to identify informed dosing strategies to control cell populations evolving under novel non-Markovian dynamics. We find that model-free deep RL is able to recover exact solutions and control cell populations even in the presence of long-range temporal dynamics. To further test our approach in more realistic settings, we demonstrate robust RL-based control strategies in environments with measurement noise and dynamic memory strength.