Multimodal Search in Chemical Documents and Reactions

TL;DR

This paper introduces a multimodal search system that enables retrieval of chemical reactions, structures, and text from scientific literature using combined visual and textual queries, enhancing chemical information access.

Contribution

The work presents a novel multimodal search tool integrating diagram extraction, reaction data parsing, and cross-modal linking for chemical literature retrieval.

Findings

Effective retrieval of chemical information from multimodal queries

Successful extraction and linking of diagrams and text in chemical documents

Positive expert assessments of system performance

Abstract

We present a multimodal search tool that facilitates retrieval of chemical reactions, molecular structures, and associated text from scientific literature. Queries may combine molecular diagrams, textual descriptions, and reaction data, allowing users to connect different representations of chemical information. To support this, the indexing process includes chemical diagram extraction and parsing, extraction of reaction data from text in tabular form, and cross-modal linking of diagrams and their mentions in text. We describe the system's architecture, key functionalities, and retrieval process, along with expert assessments of the system. This demo highlights the workflow and technical components of the search system.