MarkushGrapher-2: End-to-end Multimodal Recognition of Chemical Structures

TL;DR

MarkushGrapher-2 is an end-to-end multimodal system that accurately recognizes complex chemical Markush structures from documents by integrating OCR, vision, and layout encoding, and is supported by a large dataset and benchmark.

Contribution

It introduces a novel multimodal recognition approach for Markush structures, including a new dataset, benchmark, and a two-stage training strategy for improved accuracy.

Findings

Outperforms state-of-the-art models in Markush structure recognition.

Effectively fuses text, image, and layout information for chemical structure extraction.

Maintains strong performance in molecule recognition tasks.

Abstract

Automatically extracting chemical structures from documents is essential for the large-scale analysis of the literature in chemistry. Automatic pipelines have been developed to recognize molecules represented either in figures or in text independently. However, methods for recognizing chemical structures from multimodal descriptions (Markush structures) lag behind in precision and cannot be used for automatic large-scale processing. In this work, we present MarkushGrapher-2, an end-to-end approach for the multimodal recognition of chemical structures in documents. First, our method employs a dedicated OCR model to extract text from chemical images. Second, the text, image, and layout information are jointly encoded through a Vision-Text-Layout encoder and an Optical Chemical Structure Recognition vision encoder. Finally, the resulting encodings are effectively fused through a two-stage training strategy and used to auto-regressively generate a representation of the Markush structure. To address the lack of training data, we introduce an automatic pipeline for constructing a large-scale dataset of real-world Markush structures. In addition, we present IP5-M, a large manually-annotated benchmark of real-world Markush structures, designed to advance research on this challenging task. Extensive experiments show that our approach substantially outperforms state-of-the-art models in multimodal Markush structure recognition, while maintaining strong performance in molecule structure recognition. Code, models, and datasets are released publicly.