Transformer-CNN: Fast and Reliable tool for QSAR

TL;DR

Transformer-CNN leverages SMILES embeddings from a Transformer model combined with CharNN architecture, enabling fast, reliable, and interpretable QSAR/QSPR predictions, especially effective on small datasets.

Contribution

The paper introduces a novel Transformer-CNN approach using SMILES embeddings and augmentation, improving QSAR/QSPR modeling with interpretability and small dataset performance.

Findings

High-quality QSAR/QSPR models achieved on diverse datasets

Effective use of SMILES augmentation and transfer learning

Open-source code and online implementation available

Abstract

We present SMILES-embeddings derived from the internal encoder state of a Transformer [1] model trained to canonize SMILES as a Seq2Seq problem. Using a CharNN [2] architecture upon the embeddings results in higher quality interpretable QSAR/QSPR models on diverse benchmark datasets including regression and classification tasks. The proposed Transformer-CNN method uses SMILES augmentation for training and inference, and thus the prognosis is based on an internal consensus. That both the augmentation and transfer learning are based on embeddings allows the method to provide good results for small datasets. We discuss the reasons for such effectiveness and draft future directions for the development of the method. The source code and the embeddings needed to train a QSAR model are available on https://github.com/bigchem/transformer-cnn. The repository also has a standalone program for QSAR prognosis which calculates individual atoms contributions, thus interpreting the model's result. OCHEM [3] environment (https://ochem.eu) hosts the on-line implementation of the method proposed.