Diversity-Aware Reinforcement Learning for de novo Drug Design

TL;DR

This paper explores how different adaptive reward update mechanisms in reinforcement learning can enhance the diversity of generated drug molecules, addressing the risk of local optima in drug design models.

Contribution

It systematically investigates various intrinsic motivation and penalty strategies for reward functions to improve molecular diversity in drug generation.

Findings

Combining structure- and prediction-based methods improves diversity.

Adaptive reward mechanisms influence the diversity of generated molecules.

Diverse reward strategies lead to more promising drug candidates.

Abstract

Fine-tuning a pre-trained generative model has demonstrated good performance in generating promising drug molecules. The fine-tuning task is often formulated as a reinforcement learning problem, where previous methods efficiently learn to optimize a reward function to generate potential drug molecules. Nevertheless, in the absence of an adaptive update mechanism for the reward function, the optimization process can become stuck in local optima. The efficacy of the optimal molecule in a local optimization may not translate to usefulness in the subsequent drug optimization process or as a potential standalone clinical candidate. Therefore, it is important to generate a diverse set of promising molecules. Prior work has modified the reward function by penalizing structurally similar molecules, primarily focusing on finding molecules with higher rewards. To date, no study has comprehensively examined how different adaptive update mechanisms for the reward function influence the diversity of generated molecules. In this work, we investigate a wide range of intrinsic motivation methods and strategies to penalize the extrinsic reward, and how they affect the diversity of the set of generated molecules. Our experiments reveal that combining structure- and prediction-based methods generally yields better results in terms of diversity.