Understanding Structural Representation in Foundation Models for Polymers

TL;DR

This paper introduces a new SMILES-based polymer graph representation for foundation models, demonstrating superior performance and invariance in polymer property prediction across multiple benchmarks.

Contribution

The paper presents a novel polymer graph representation method using SMILES, improving foundation model performance and providing insights into representation invariance and model behavior.

Findings

The new representation outperforms other variations in benchmark tests.

SMILES-based models show high invariance, with many variations achieving near state-of-the-art results.

Models interpolate over sequence space, including invalid inputs, affecting prediction accuracy.

Abstract

From the relative scarcity of training data to the lack of standardized benchmarks, the development of foundation models for polymers face significant and multi-faceted challenges. At the core, many of these issues are tied directly to the structural representation of polymers and here, we present a new foundation model using a SMILES-based polymer graph representation. This approach allows representation of critical polymer architectural features and connectivity that are not available in other SMILES-based representations. The developed polymer foundation model exhibited excellent performance on 28 different benchmark datasets. Critical evaluation of the developed representation against other variations in control experiments reveals this approach to be a highly performant method of representing polymers in language-based foundation models. These control experiments also reveal a strong invariance of all SMILES representations, with many variations achieving state-of-the-art or near state-of-the-art performance, including those which are chemically or semantically invalid. Examination of error sources and attention maps for the evaluated representations corroborate the findings of the control experiments, showing that chemistry language models based on SMILES interpolate over all sequence space for prediction tasks, not only those of semantically valid inputs. Overall, this work highlights the importance of control experiments as a check on human-imposed assumptions that can limit rational design of both chemistry foundation models and their underlying structural representations.