# Multimodal pre-training models of molecular representation for drug discovery

## TL;DR

This paper reviews how multimodal pre-training models are being used to improve drug discovery by combining different data types and techniques.

## Contribution

The paper systematically reviews multimodal pre-training models and identifies emerging trends in molecular representation for drug discovery.

## Key findings

- Multimodal pre-training models integrate Transformers and graph neural networks for cross-scale molecular representation.
- Molecular captions help bridge drug discovery with large language models.
- Adaptability between modalities and pre-training tasks is crucial for model effectiveness.

## Abstract

With the great success of large language models in natural language processing, self-supervised pre-training models have emerged as an important technique in drug discovery. In particular, multimodal pre-training models have opened a new avenue for drug discovery. The experience and ideas from previous works can provide important reference points for further research in drug discovery. Therefore, this review summarizes the foundation of multimodal pre-training models and their progress in the field of drug discovery. We emphasize the adaptability between various modalities and network frameworks or pre-training tasks. At the same time, we summarize the difference and relevance between various modalities or pre-training models. Importantly, we identify two increasing trends that may serve as reference points for future research. Specifically, Transformers and graph neural networks are often integrated as encoders and then combined with multiple pre-training tasks to learn cross-scale molecular representation, thereby promoting the accuracy of drug discovery. In addition, molecular captions as brief biomedical text provide a bridge for collaboration between drug discovery and large language models. Finally, we discuss the challenges of multimodal pre-training models in drug discovery, and explore future opportunities.

A systematic review summarizing the foundation of multimodal pre-training models and their progress in molecular representation of drug discovery.