MolVision: Molecular Property Prediction with Vision Language Models

TL;DR

MolVision introduces a multimodal approach combining molecular images and text using vision-language models to improve molecular property prediction across diverse datasets.

Contribution

This work pioneers the integration of visual molecular representations with textual descriptions in VLMs for property prediction, demonstrating significant performance gains.

Findings

Visual information enhances prediction accuracy.

Multimodal fusion improves generalization.

Fine-tuning with LoRA boosts performance.

Abstract

Molecular property prediction is a fundamental task in computational chemistry with critical applications in drug discovery and materials science. While recent works have explored Large Language Models (LLMs) for this task, they primarily rely on textual molecular representations such as SMILES/SELFIES, which can be ambiguous and structurally less informative. In this work, we introduce MolVision, a novel approach that leverages Vision-Language Models (VLMs) by integrating both molecular structure as images and textual descriptions to enhance property prediction. We construct a benchmark spanning ten diverse datasets, covering classification, regression and description tasks. Evaluating nine different VLMs in zero-shot, few-shot, and fine-tuned settings, we find that visual information improves prediction performance, particularly when combined with efficient fine-tuning strategies such as LoRA. Our results reveal that while visual information alone is insufficient, multimodal fusion significantly enhances generalization across molecular properties. Adaptation of vision encoder for molecular images in conjunction with LoRA further improves the performance. The code and data is available at : $\href{https://molvision.github.io/MolVision/}{https://molvision.github.io/MolVision/}$.