Chemical transformer compression for accelerating both training and inference of molecular modeling

TL;DR

This paper introduces DeLiCaTe, a compressed chemical transformer model that significantly accelerates training and inference in molecular modeling while maintaining high predictive performance.

Contribution

It presents a novel combination of cross-layer parameter sharing and knowledge distillation to create a lightweight transformer for molecular science.

Findings

DeLiCaTe achieves 4x faster training and inference.

It reduces parameters by 10 times and layers by 3 times.

Maintains comparable QSAR and virtual screening performance.

Abstract

Transformer models have been developed in molecular science with excellent performance in applications including quantitative structure-activity relationship (QSAR) and virtual screening (VS). Compared with other types of models, however, they are large, which results in a high hardware requirement to abridge time for both training and inference processes. In this work, cross-layer parameter sharing (CLPS), and knowledge distillation (KD) are used to reduce the sizes of transformers in molecular science. Both methods not only have competitive QSAR predictive performance as compared to the original BERT model, but also are more parameter efficient. Furthermore, by integrating CLPS and KD into a two-state chemical network, we introduce a new deep lite chemical transformer model, DeLiCaTe. DeLiCaTe captures general-domains as well as task-specific knowledge, which lead to a 4x faster rate of both training and inference due to a 10- and 3-times reduction of the number of parameters and layers, respectively. Meanwhile, it achieves comparable performance in QSAR and VS modeling. Moreover, we anticipate that the model compression strategy provides a pathway to the creation of effective generative transformer models for organic drug and material design.