Pre-training Transformers for Molecular Property Prediction Using Reaction Prediction

TL;DR

This paper introduces a pre-training method for molecular property prediction using reaction data to improve the performance of a SMILES Transformer model across multiple tasks, demonstrating the effectiveness of transfer learning in chemistry.

Contribution

It presents a novel pre-training procedure for molecular representations using reaction data, enhancing transfer learning for molecular property prediction.

Findings

Significant improvement on 5 out of 12 tasks with pre-training.

Pre-training with reaction data benefits molecular property prediction.

The approach outperforms non-pre-trained baselines on several benchmarks.

Abstract

Molecular property prediction is essential in chemistry, especially for drug discovery applications. However, available molecular property data is often limited, encouraging the transfer of information from related data. Transfer learning has had a tremendous impact in fields like Computer Vision and Natural Language Processing signaling for its potential in molecular property prediction. We present a pre-training procedure for molecular representation learning using reaction data and use it to pre-train a SMILES Transformer. We fine-tune and evaluate the pre-trained model on 12 molecular property prediction tasks from MoleculeNet within physical chemistry, biophysics, and physiology and show a statistically significant positive effect on 5 of the 12 tasks compared to a non-pre-trained baseline model.