A quantitative analysis of knowledge-learning preferences in large language models in molecular science

TL;DR

This paper introduces a multi-modal benchmark and analysis framework to quantify how large language models learn and adapt to different data modalities in molecular science, enhancing understanding of their knowledge acquisition.

Contribution

It proposes a novel benchmark and a statistical approach to analyze modality compatibility and knowledge preferences in large language models for molecular applications.

Findings

Identified key data modalities for specific molecular tasks

Developed a statistically interpretable method for knowledge mapping

Provided insights into model-data modality compatibility

Abstract

Deep learning has significantly advanced molecular modeling and design, enabling efficient understanding and discovery of novel molecules. In particular, large language models (LLMs) introduce a fresh research paradigm to tackle scientific problems from a natural language processing (NLP) perspective. LLMs significantly enhance our understanding and generation of molecules, often surpassing existing methods with their capabilities to decode and synthesize complex molecular patterns. However, two key issues remain: how to quantify the match between model and data modalities and how to identify the knowledge-learning preferences of models. To address these challenges, we propose a multi-modal benchmark, named ChEBI-20-MM, and perform 1263 experiments to assess the model's compatibility with data modalities and knowledge acquisition. Through the modal transition probability matrix, we provide insights into the most suitable modalities for tasks. Furthermore, we introduce a statistically interpretable approach to discover context-specific knowledge mapping by localized feature filtering. Our analysis offers an exploration of the learning mechanism and paves the way for advancing LLMs in molecular science.