Reinforcement Learning with LLM-Guided Action Spaces for Synthesizable Lead Optimization

TL;DR

MolReAct introduces a reinforcement learning framework guided by large language models that optimizes drug lead molecules within a chemically valid, synthesis-constrained action space, achieving high property scores efficiently.

Contribution

This work presents MolReAct, a novel method combining LLM-guided action spaces with reaction templates and policy optimization for synthesizable lead optimization.

Findings

Achieved the highest average Top-10 score of 0.571 across multiple tasks.

Reduced optimization time by approximately 43% using SMILES caching.

Outperformed all baselines on 13 out of 14 property optimization tasks.

Abstract

Lead optimization in drug discovery requires improving therapeutic properties while ensuring that molecular modifications correspond to feasible synthetic routes. Existing approaches either prioritize property scores without enforcing synthesizability, or rely on expensive enumeration over large reaction networks, while direct application of Large Language Models (LLMs) to molecular generation frequently produces chemically invalid structures. We introduce MolReAct, a framework that formulates lead optimization as a Markov Decision Process over a synthesis-constrained action space defined by validated reaction templates. A tool-augmented LLM agent serves as a dynamic reaction environment, invoking specialized chemical analysis tools to identify reactive sites and functional groups and proposing a compact set of chemically grounded transformations from matched templates. A dedicated policy model trained via Group Relative Policy Optimization (GRPO) selects among these constrained actions to maximize long-term oracle reward across multi-step trajectories, with a SMILES-based caching mechanism reducing end-to-end optimization time by approximately 43%. Across 13 property optimization tasks from the Therapeutic Data Commons and one structure-based docking task, MolReAct achieves an average Top-10 score of 0.571, the highest among all baselines, ranking first or second on 13 of 14 tasks and attaining the best sample efficiency on 9 of 14 tasks. By grounding every optimization step in validated reaction templates, MolReAct produces molecules that are not only property-improved but each accompanied by an explicit template-grounded synthetic pathway.