Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representation

TL;DR

This paper systematically investigates how molecular language models scale with size, data, and representation, revealing predictable scaling laws and the significant influence of molecular representation on performance.

Contribution

It provides the first comprehensive analysis of scaling behaviors in molecular language models, including a large-scale experimental study and public release of models and code.

Findings

Clear scaling laws for molecular models in pretraining and transfer tasks

Molecular representation significantly impacts model performance

Reveals reasons for previous inconsistencies in molecular generation scaling

Abstract

Molecular generative models, often employing GPT-style language modeling on molecular string representations, have shown promising capabilities when scaled to large datasets and model sizes. However, it remains unclear and subject to debate whether these models adhere to predictable scaling laws under fixed computational budgets, which is a crucial understanding for optimally allocating resources between model size, data volume, and molecular representation. In this study, we systematically investigate the scaling behavior of molecular language models across both pretraining and downstream tasks. We train 300 models and conduct over 10,000 experiments, rigorously controlling compute budgets while independently varying model size, number of training tokens, and molecular representation. Our results demonstrate clear scaling laws in molecular models for both pretraining and downstream transfer, reveal the substantial impact of molecular representation on performance, and explain previously observed inconsistencies in scaling behavior for molecular generation. Additionally, we publicly release the largest library of molecular language models to date to facilitate future research and development. Code and models are available at https://github.com/SZU-ADDG/MLM-Scaling.