HIGHT: Hierarchical Graph Tokenization for Molecule-Language Alignment

TL;DR

HIGHT introduces a hierarchical graph tokenization method that captures molecular structures at multiple levels, significantly improving molecule-language alignment and reducing hallucinations in large language models.

Contribution

The paper proposes HIGHT, a novel hierarchical graph tokenizer that encodes atom, motif, and molecular levels, enhancing LLM perception of molecules and addressing limitations of prior flat tokenization methods.

Findings

Reduces hallucination by 40% in molecule-language tasks

Improves performance across 14 real-world benchmarks

Enhances molecular understanding in LLMs

Abstract

Recently, there has been a surge of interest in extending the success of large language models (LLMs) from texts to molecules. Most existing approaches adopt a graph neural network to represent a molecule as a series of node tokens for molecule-language alignment, which, however, have overlooked the inherent hierarchical structures in molecules. Notably, higher-order molecular structures contain rich semantics of functional groups, which encode crucial biochemical functionalities of the molecules. We show that neglecting the hierarchical information in tokenization will lead to subpar molecule-language alignment and severe hallucination. To address this limitation, we propose HIerarchical GrapH Tokenization (HIGHT). HIGHT employs a hierarchical graph tokenizer that encodes the hierarchy of atom, motif, and molecular levels of informative tokens to improve the molecular perception of LLMs. HIGHT also adopts an augmented instruction tuning dataset, enriched with the hierarchical graph information, to further enhance the molecule-language alignment. Extensive experiments on 14 real-world benchmarks verify the effectiveness of HIGHT in reducing hallucination by 40%, and significant improvements in various molecule-language downstream tasks. The project is available at https: //higraphllm.github.io/.