Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB files

TL;DR

This paper demonstrates that language models trained on 3D chemical file formats can directly generate molecules, materials, and protein binding sites in three dimensions without architectural modifications, matching state-of-the-art performance.

Contribution

It shows that language models can generate 3D molecular structures directly from raw file formats, expanding the scope beyond graph-based representations.

Findings

Language models trained on XYZ, CIF, and PDB files can generate valid 3D structures.

Performance is comparable to specialized graph-based and domain-specific models.

No architectural modifications are needed for 3D structure generation.

Abstract

Language models are powerful tools for molecular design. Currently, the dominant paradigm is to parse molecular graphs into linear string representations that can easily be trained on. This approach has been very successful, however, it is limited to chemical structures that can be completely represented by a graph -- like organic molecules -- while materials and biomolecular structures like protein binding sites require a more complete representation that includes the relative positioning of their atoms in space. In this work, we show how language models, without any architecture modifications, trained using next-token prediction -- can generate novel and valid structures in three dimensions from various substantially different distributions of chemical structures. In particular, we demonstrate that language models trained directly on sequences derived directly from chemical file formats like XYZ files, Crystallographic Information files (CIFs), or Protein Data Bank files (PDBs) can directly generate molecules, crystals, and protein binding sites in three dimensions. Furthermore, despite being trained on chemical file sequences -- language models still achieve performance comparable to state-of-the-art models that use graph and graph-derived string representations, as well as other domain-specific 3D generative models. In doing so, we demonstrate that it is not necessary to use simplified molecular representations to train chemical language models -- that they are powerful generative models capable of directly exploring chemical space in three dimensions for very different structures.