Generalist Large Language Models for Molecular Property Prediction: Distilling Knowledge from Specialist Models

TL;DR

This paper introduces TreeKD, a knowledge distillation method that enhances large language models for molecular property prediction by incorporating rule-based knowledge from decision trees, significantly improving their performance.

Contribution

The paper presents TreeKD, a novel approach that distills knowledge from tree-based specialist models into LLMs using verbalized rules, improving their predictive accuracy in molecular property tasks.

Findings

TreeKD improves LLM performance on ADMET properties.

It narrows the gap between generalist LLMs and specialist models.

Rule-consistency enhances prediction stability.

Abstract

Molecular Property Prediction (MPP) is a central task in drug discovery. While Large Language Models (LLMs) show promise as generalist models for MPP, their current performance remains below the threshold for practical adoption. We propose TreeKD, a novel knowledge distillation method that transfers complementary knowledge from tree-based specialist models into LLMs. Our approach trains specialist decision trees on functional group features, then verbalizes their learned predictive rules as natural language to enable rule-augmented context learning. This enables LLMs to leverage structural insights that are difficult to extract from SMILES strings alone. We further introduce rule-consistency, a test-time scaling technique inspired by bagging that ensembles predictions across diverse rules from a Random Forest. Experiments on 22 ADMET properties from the TDC benchmark demonstrate that TreeKD substantially improves LLM performance, narrowing the gap with SOTA specialist models and advancing toward practical generalist models for molecular property prediction.