Molecular representation learning with language models and domain-relevant auxiliary tasks

TL;DR

This paper demonstrates that using domain-relevant auxiliary tasks in pre-training Transformer models like BERT significantly enhances molecular representations for drug discovery, outperforming existing methods on key benchmarks.

Contribution

The study introduces MolBert, a Transformer-based model trained with domain-specific auxiliary tasks, improving molecular representation quality for virtual screening and QSAR tasks.

Findings

Auxiliary tasks with domain relevance improve downstream performance

Choice of self-supervised tasks significantly affects results

MolBert outperforms current state-of-the-art models on benchmarks

Abstract

We apply a Transformer architecture, specifically BERT, to learn flexible and high quality molecular representations for drug discovery problems. We study the impact of using different combinations of self-supervised tasks for pre-training, and present our results for the established Virtual Screening and QSAR benchmarks. We show that: i) The selection of appropriate self-supervised task(s) for pre-training has a significant impact on performance in subsequent downstream tasks such as Virtual Screening. ii) Using auxiliary tasks with more domain relevance for Chemistry, such as learning to predict calculated molecular properties, increases the fidelity of our learnt representations. iii) Finally, we show that molecular representations learnt by our model `MolBert' improve upon the current state of the art on the benchmark datasets.