MolNexTR: A Generalized Deep Learning Model for Molecular Image Recognition

TL;DR

MolNexTR is a novel deep learning model combining ConvNext and Vision-Transformer to accurately convert diverse molecular images into machine-readable formats, advancing chemical structure recognition.

Contribution

It introduces a generalized image-to-graph model that integrates local and global feature extraction with symbolic chemistry principles for improved molecular image recognition.

Findings

Achieved 81-97% accuracy on test sets.

Outperformed existing methods in recognizing diverse molecular images.

Enhanced robustness through advanced data augmentation and contamination modules.

Abstract

In the field of chemical structure recognition, the task of converting molecular images into machine-readable data formats such as SMILES string stands as a significant challenge, primarily due to the varied drawing styles and conventions prevalent in chemical literature. To bridge this gap, we proposed MolNexTR, a novel image-to-graph deep learning model that collaborates to fuse the strengths of ConvNext, a powerful Convolutional Neural Network variant, and Vision-TRansformer. This integration facilitates a more detailed extraction of both local and global features from molecular images. MolNexTR can predict atoms and bonds simultaneously and understand their layout rules. It also excels at flexibly integrating symbolic chemistry principles to discern chirality and decipher abbreviated structures. We further incorporate a series of advanced algorithms, including an improved data augmentation module, an image contamination module, and a post-processing module for getting the final SMILES output. These modules cooperate to enhance the model's robustness to diverse styles of molecular images found in real literature. In our test sets, MolNexTR has demonstrated superior performance, achieving an accuracy rate of 81-97%, marking a significant advancement in the domain of molecular structure recognition.