Enhancing reinforcement learning for population setpoint tracking in co-cultures

TL;DR

This paper introduces a novel reinforcement learning approach with a specialized return function to improve population setpoint tracking in co-cultures, demonstrated on an E. coli chemostat with optogenetic control.

Contribution

It proposes a new return function for policy-gradient reinforcement learning that better guides multi-setpoint tracking in biotechnological co-cultures.

Findings

Enhanced tracking accuracy with the new return function

Improved control performance in E. coli co-culture

Demonstrated robustness across different setpoints

Abstract

Efficient multiple setpoint tracking can enable advanced biotechnological applications, such as maintaining desired population levels in co-cultures for optimal metabolic division of labor. In this study, we employ reinforcement learning as a control method for population setpoint tracking in co-cultures, focusing on policy-gradient techniques where the control policy is parameterized by neural networks. However, achieving accurate tracking across multiple setpoints is a significant challenge in reinforcement learning, as the agent must effectively balance the contributions of various setpoints to maximize the expected system performance. Traditional return functions, such as those based on a quadratic cost, often yield suboptimal performance due to their inability to efficiently guide the agent toward the simultaneous satisfaction of all setpoints. To overcome this, we propose a novel return function that rewards the simultaneous satisfaction of multiple setpoints and diminishes overall reward gains otherwise, accounting for both stage and terminal system performance. This return function includes parameters to fine-tune the desired smoothness and steepness of the learning process. We demonstrate our approach considering an $\textit{Escherichia coli}$ co-culture in a chemostat with optogenetic control over amino acid synthesis pathways, leveraging auxotrophies to modulate growth.