How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?

TL;DR

This study systematically evaluates how increasing training resources for chemical language models affects their transfer performance on molecular property prediction tasks, revealing limited improvements despite better pretraining metrics.

Contribution

It challenges the assumption that larger models and datasets automatically lead to better downstream performance in chemical language modeling.

Findings

Pretraining loss decreases with more resources, but downstream performance plateaus.

Alternative metrics like Hessian-based measures do not predict downstream success.

Downstream performance can saturate or degrade even as pretraining metrics improve.

Abstract

Chemical Language Models (CLMs) pre-trained on large scale molecular data are widely used for molecular property prediction. However, the common belief that increasing training resources such as model size, dataset size, and training compute improves both pretraining loss and downstream task performance has not been systematically validated in the chemical domain. In this work, we evaluate this assumption by pretraining CLMs while scaling training resources and measuring transfer performance across diverse molecular property prediction (MPP) tasks. We find that while pretraining loss consistently decreases with increased training resources, downstream task performance shows limited improvement. Moreover, alternative metrics based on the Hessian or loss landscape also fail to estimate downstream performance in CLMs. We further identify conditions under which downstream performance saturates or degrades despite continued improvements in pretraining metrics, and analyze the underlying task dependent failure modes through parameter space visualizations. These results expose a gap between pretraining based evaluation and downstream performance, and emphasize the need for model selection and evaluation strategies that explicitly account for downstream task characteristics.