Improving Large Molecular Language Model via Relation-aware Multimodal Collaboration

TL;DR

This paper introduces CoLLaMo, a relation-aware multimodal large language model for molecules that improves integration of diverse molecular data, reducing hallucinations and enhancing performance across various molecular understanding tasks.

Contribution

The paper proposes a novel relation-aware multimodal attention mechanism and an automatic evaluation framework, advancing molecular language models' robustness and multimodal integration capabilities.

Findings

Achieves state-of-the-art results on multiple molecular tasks

Reduces hallucination and improves robustness in molecular understanding

Enhances generalization across molecular modalities

Abstract

Large language models (LLMs) have demonstrated their instruction-following capabilities and achieved powerful performance on various tasks. Inspired by their success, recent works in the molecular domain have led to the development of large molecular language models (LMLMs) that integrate 1D molecular strings or 2D molecular graphs into the language models. However, existing LMLMs often suffer from hallucination and limited robustness, largely due to inadequate integration of diverse molecular modalities such as 1D sequences, 2D molecular graphs, and 3D conformations. To address these limitations, we propose CoLLaMo, a large language model-based molecular assistant equipped with a multi-level molecular modality-collaborative projector. The relation-aware modality-collaborative attention mechanism in the projector facilitates fine-grained and relation-guided information exchange between atoms by incorporating 2D structural and 3D spatial relations. Furthermore, we present a molecule-centric new automatic measurement, including a hallucination assessment metric and GPT-based caption quality evaluation to address the limitations of token-based generic evaluation metrics (i.e., BLEU) widely used in assessing molecular comprehension of LMLMs. Our extensive experiments demonstrate that our CoLLaMo enhances the molecular modality generalization capabilities of LMLMs, achieving the best performance on multiple tasks, including molecule captioning, computed property QA, descriptive property QA, motif counting, and IUPAC name prediction.