Benchmarking Large Language Models for Polymer Property Predictions

TL;DR

This study benchmarks large language models like LLaMA-3 and GPT-3.5 for predicting polymer thermal properties, comparing their performance to traditional methods and analyzing their strengths and limitations in polymer informatics.

Contribution

It introduces a systematic benchmarking of open-source and commercial LLMs for polymer property prediction, highlighting their current capabilities and limitations compared to traditional fingerprinting approaches.

Findings

LLMs approach but do not surpass traditional models in accuracy

LLaMA-3 outperforms GPT-3.5 due to open-source flexibility

Single-task learning is more effective than multi-task for LLMs in this context

Abstract

Machine learning has revolutionized polymer science by enabling rapid property prediction and generative design. Large language models (LLMs) offer further opportunities in polymer informatics by simplifying workflows that traditionally rely on large labeled datasets, handcrafted representations, and complex feature engineering. LLMs leverage natural language inputs through transfer learning, eliminating the need for explicit fingerprinting and streamlining training. In this study, we finetune general purpose LLMs -- open-source LLaMA-3-8B and commercial GPT-3.5 -- on a curated dataset of 11,740 entries to predict key thermal properties: glass transition, melting, and decomposition temperatures. Using parameter-efficient fine-tuning and hyperparameter optimization, we benchmark these models against traditional fingerprinting-based approaches -- Polymer Genome, polyGNN, and polyBERT -- under single-task (ST) and multi-task (MT) learning. We find that while LLM-based methods approach traditional models in performance, they generally underperform in predictive accuracy and efficiency. LLaMA-3 consistently outperforms GPT-3.5, likely due to its tunable open-source architecture. Additionally, ST learning proves more effective than MT, as LLMs struggle to capture cross-property correlations, a key strength of traditional methods. Analysis of molecular embeddings reveals limitations of general purpose LLMs in representing nuanced chemo-structural information compared to handcrafted features and domain-specific embeddings. These findings provide insight into the interplay between molecular embeddings and natural language processing, guiding LLM selection for polymer informatics.