Faster and more diverse de novo molecular optimization with double-loop reinforcement learning using augmented SMILES

TL;DR

This paper introduces a double-loop reinforcement learning method with SMILES augmentation to accelerate molecular optimization, enhance diversity, and improve reproducibility in generative models for drug discovery.

Contribution

It proposes a novel double-loop reinforcement learning approach using SMILES augmentation to increase efficiency and diversity in molecular generation tasks.

Findings

Optimal augmentation repetitions are between 5 and 10.

The method increases diversity and similarity to known ligands.

It speeds up learning by reusing scoring calculations.

Abstract

Using generative deep learning models and reinforcement learning together can effectively generate new molecules with desired properties. By employing a multi-objective scoring function, thousands of high-scoring molecules can be generated, making this approach useful for drug discovery and material science. However, the application of these methods can be hindered by computationally expensive or time-consuming scoring procedures, particularly when a large number of function calls are required as feedback in the reinforcement learning optimization. Here, we propose the use of double-loop reinforcement learning with simplified molecular line entry system (SMILES) augmentation to improve the efficiency and speed of the optimization. By adding an inner loop that augments the generated SMILES strings to non-canonical SMILES for use in additional reinforcement learning rounds, we can both reuse the scoring calculations on the molecular level, thereby speeding up the learning process, as well as offer additional protection against mode collapse. We find that employing between 5 and 10 augmentation repetitions is optimal for the scoring functions tested and is further associated with an increased diversity in the generated compounds, improved reproducibility of the sampling runs and the generation of molecules of higher similarity to known ligands.