Judging Chemical Reaction Practicality From Positive Sample only Learning

TL;DR

This paper introduces a machine learning approach that predicts the practicality of chemical reactions using only positive samples, achieving high accuracy without relying on complex chemistry knowledge.

Contribution

It proposes an auto-construction method to address the negative sample insufficiency problem in reaction practicality prediction.

Findings

Achieves 99.76% accuracy on large-scale reaction data

Effectively predicts reaction practicality without negative samples

Introduces a novel auto-construction method for training data

Abstract

Chemical reaction practicality is the core task among all symbol intelligence based chemical information processing, for example, it provides indispensable clue for further automatic synthesis route inference. Considering that chemical reactions have been represented in a language form, we propose a new solution to generally judge the practicality of organic reaction without considering complex quantum physical modeling or chemistry knowledge. While tackling the practicality judgment as a machine learning task from positive and negative (chemical reaction) samples, all existing studies have to carefully handle the serious insufficiency issue on the negative samples. We propose an auto-construction method to well solve the extensively existed long-term difficulty. Experimental results show our model can effectively predict the practicality of chemical reactions, which achieves a high accuracy of 99.76\% on real large-scale chemical lab reaction practicality judgment.

Figures

Tables

Table 1. Table 1: Examples of SMILES

SMILES	Name	SMILES	Name
CC	ethane	[OH3+]	hydronium ion
O=C=O	carbon dioxide	[2H]O[2H]	deuterium oxide
C#N	hydrogen cyanide	[235U]	uranium-235
CCN(CC)CC	triethylamine	F/C=C/F	E-difluoroethene
CC(=O)O	acetic acid	F/C=C\F	Z-difluoroethene
C1CCCCC1	cyclohexane	N[C@@H](C)C(=O)O	L-alanine
c1ccccc1	benzene	N[C@H](C)C(=O)O	D-alanine

Table 2. Table 2: Data processing steps. The tokens are separated by a space and individual molecules by a point token.

Step	Example (reactants $>$ reagents $>$ products)
$1)$ Original string	[C:1]([C:3]1[CH:8]=[CH:7][CH:6]=[CH:5][C:4]=1 [OH:9])#[N:2].[CH2:10]([CH:12]1[O:14][CH2:13] 1)Cl>N1CCCCC1>[O:14]1[CH2:13][CH:12]1[CH2:10] [O:9][C:4]1[CH:5]=[CH:6][CH:7]=[CH:8][C:3]=1 [C:1]#[N:2]
$2)$ Atom-mapping removal and canonicalization	ClCC1CO1.N#Cc1ccccc1O>N1CCCCC1>N#Cc1ccccc1OC C1CO1
$3)$ Tokenization atom-wise	Cl C C 1 C O 1 . N # C c 1 c c c c c 1 O > N 1 C C C C C 1 > N # C c 1 c c cc c 1 O C C 1 C O 1
$4)$ DLG segmentation	ClC C 1 CO1. N#Cc 1cccc c1 O > N1CC CCC 1> N#Cc 1cccc c1 O C C 1 C O 1

Table 3. Table 3: Distributions of positive and negative reaction from the train/dev/test sets in four combinations.

Data	Case	train	dev	test
CJHIF + Chemical.AI-Real-1	Positive	1,406,259	156,251	173,624
	Negative	7,178	798	874
USPTO + Chemical.AI-Real-1	Positive	217,992	24,221	26,919
	Negative	7,176	797	877
CJHIF + Chemical.AI-Rule	Positive	1,428,673	158,741	89,948
	Negative	158,689	17,632	10,052
USPTO + Chemical.AI-Rule	Positive	217,799	24,200	90,221
	Negative	24,421	2,713	9,779
Chemical.AI-Real-2	Positive	8,069	-	8,082
	Negative	4,202	-	4,175

Table 4. Table 4: Possible prediction results

	Predicted Positive	Predicted Negative
True Positive	TP	FN
True Negative	FP	TN

Table 5. Table 5: Examples of SMILES words from DLG segmentation

Word	DLG Score	Word	DLG Score
[Rh]789%10	78.18	[Ru++]5678	74.61
Mo+6]89%10	68.77	3[Zn++]579	66.74
Mg]Br)cc1.	52.68	ccc3)[Ru++	49.01
C#C[Mg]Br	47.60	C=OBr[Mg]	40.02
\C=C/I	20.09	(C#N)	7.68

Table 6. Table 6: Ablation study for practicality judgment(F1 Score on Positive case / Negative case) (%)

		USPTO + Chemical.AI-Real-1				USPTO + Chemical.AI-Rule
Features	Case	Precision	Recall	F1	Acc	Precision	Recall	F1	Acc
Full	P	98.83	99.21	99.02	97.92	97.89	96.72	97.30	96.26
	N	72.45	63.83	67.88		91.47	93.94	92.68
w/o RSD	P	98.81	98.80	98.81	97.78	97.21	96.54	96.87	95.62
	N	63.27	63.63	63.45		90.58	92.31	91.44
w/o Tokenization	P	98.58	99.01	98.79	97.53	96.69	92.33	94.46	93.34
	N	64.87	56.21	60.23		84.56	91.53	86.31
w/o RSD &	P	98.68	98.61	98.65	97.32	95.75	93.58	94.65	93.31
Tokenization	N	58.28	59.41	58.84		83.77	88.87	86.24

Table 7. Table 7: Comparison of F1-score for practicality judgment on USPTO + Chemical.AI-Real-1(%)

Model	Case	Precision	Recall	F1-score	Accuracy
Siamese	Positive	98.83	99.21	99.02	97.92
	Negative	72.45	63.83	67.88
LSTM	Positive	98.78	98.92	98.85	97.77
	Negative	65.24	60.49	63.83
BiLSTM	Positive	98.87	99.04	98.96	97.97
	Negative	68.87	65.34	67.06
GRU	Positive	99.13	98.53	98.83	97.74
	Negative	61.92	73.43	67.19
BiGRU	Positive	98.83	99.18	99.01	97.93
	Negative	71.78	64.08	67.71

Table 8. Table 8: Results for practicality judgment (%)

Data	Case	Precision	Recall	F1-score	Accuracy
CJHIF + Chemical.AI-Real-1	P	99.82	99.95	99.88	99.76
	N	86.09	63.73	72.24
USPTO + Chemical.AI-Real-1	P	98.83	99.21	99.02	97.92
	N	72.45	63.83	67.88
CJHIF + Chemical.AI-Rule	P	96.19	99.91	98.02	97.96
	N	95.15	75.98	84.49
USPTO + Chemical.AI-Rule	P	97.89	96.72	97.30	98.97
	N	91.47	93.94	92.68

Table 9. Table 9: Results for practicality judgment in Chemical.AI-Real-2 dataset (%)

Training Data	Case	Precision	Recall	F1-score	Accuracy
CJHIF + Chemical.AI-Real-1	P	66.00	85.81	74.61	61.49
	N	34.42	14.42	20.32
USPTO + Chemical.AI-Real-1	P	66.19	26.31	37.65	42.98
	N	37.65	34.15	73.99
CJHIF + Chemical.AI-Rule	P	72.03	79.05	75.38	64.75
	N	50.01	40.57	44.80
USPTO + Chemical.AI-Rule	P	70.03	66.15	68.03	60.31
	N	42.64	41.58	42.10

Equations

$$\{w^{\ast },t^{\ast }\}={\mathrm{argmax}}_{w_1\dots w_i\dots w_n=T}\sum _{i=1}^{n}g(w_i)$$

$$g_{DLG}(x_{i..j})=L(X)-L(X[r\to x_{i..j}]\oplus x_{i..j}),$$

$$L(X)\doteq -|X|\sum _{x\in V}\hat{p}(x){\log }_2\hat{p}(x),$$

$${\mathbf{M}}_1=\left[\begin{array}{c}{\mathbf{R}}_1^1\oplus {\mathbf{S}}_1^1\\ ⋮\\ {\mathbf{R}}_h^1\oplus {\mathbf{S}}_h^1\end{array}\right],{\mathbf{M}}_2=\left[\begin{array}{c}{\mathbf{P}}_1^1\oplus {\mathbf{S}}_1^1\\ ⋮\\ {\mathbf{P}}_h^1\oplus {\mathbf{S}}_h^1\end{array}\right]$$

$${\mathbf{i}}_{\mathbf{t}}=\sigma ({\mathbf{W}}_w^i{\mathbf{x}}_t+{\mathbf{W}}_h^i{\mathbf{h}}_{t-1}+{\mathbf{b}}_i),$$

$${\mathbf{f}}_{\mathbf{t}}=\sigma ({\mathbf{W}}_w^f{\mathbf{x}}_t+{\mathbf{W}}_h^f{\mathbf{h}}_{t-1}+{\mathbf{b}}_f),$$

$${\mathbf{u}}_{\mathbf{t}}=\sigma ({\mathbf{W}}_w^u{\mathbf{x}}_t+{\mathbf{W}}_h^u{\mathbf{h}}_{t-1}+{\mathbf{b}}_u),$$

$${\mathbf{c}}_{\mathbf{t}}={\mathbf{f}}_t\odot {\mathbf{c}}_{t-1}+{\mathbf{i}}_t\odot \tanh ({\mathbf{W}}_w^c{\mathbf{x}}_t+{\mathbf{W}}_h^c{\mathbf{h}}_{t-1}+{\mathbf{b}}_c),$$

$${\mathbf{h}}_{\mathbf{t}}=\tanh ({\mathbf{c}}_t)\odot {\mathbf{u}}_t,$$

$$\mathbf{y}=\frac{1}{1+e^{\mathbf{x}}}$$

$$\mathcal{L}=-\frac{1}{N}\sum _{t=1}^n\left[{\mathbf{y}}_t\mathbf{log}{\hat{\mathbf{y}}}_t+(1-{\mathbf{y}}_t)\mathbf{log}(1-{\hat{\mathbf{y}}}_t)\right]$$

$$=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+\text{FN}+\text{FP}},$$

$$=\frac{\text{TP}}{\text{TP}+\text{FP}},$$

$$=\frac{\text{TP}}{\text{TP}+\text{FN}},$$

$$=\frac{2\times \text{Precision}\times \text{Recall}}{\text{Precision}+\text{Recall}}$$

Full text

Judging Chemical Reaction Practicality From Positive Sample only Learning

Shu Jiang,1,2∗ Zhuosheng Zhang,1,2∗

Hai Zhao,1,2† Jiangtong Li,2,3 Yang Yang,1,2 Bao-Liang Lu,1,2 Ning Xia4

1Department of Computer Science and Engineering, Shanghai Jiao Tong University,

Shanghai, 200240, China

2Key Laboratory of Shanghai Education Commission for Intelligent Interaction,

and Cognitive Engineering, Shanghai Jiao Tong University,

Shanghai, 200240, China

3College of Zhiyuan, Shanghai Jiao Tong University,

Shanghai, 200240, China

4Chemical.AI, Shanghai, 200240, China

∗These authors contributed equally to this work.

†Corresponding author. Email: zhaohai@cs.sjtu.edu.cn

Chemical reaction practicality is the core task among all symbol intelligence based chemical information processing, for example, it provides indispensable clue for further automatic synthesis route inference. Considering that chemical reactions have been represented in a language form, we propose a new solution to generally judge the practicality of organic reaction without considering complex quantum physical modeling or chemistry knowledge. While tackling the practicality judgment as a machine learning task from positive and negative (chemical reaction) samples, all existing studies have to carefully handle the serious insufficiency issue on the negative samples. We propose an auto-construction method to well solve the extensively existed long-term difficulty. Experimental results show our model can effectively predict the practicality of chemical reactions, which achieves a high accuracy of 99.76% on real large-scale chemical lab reaction practicality judgment.

INTRODUCTION

Organic reactions including addition reactions  (?), elimination reaction  (?), substitution reactions  (?, ?, ?), pericyclic reactions  (?), rearrangement reactions  (?, ?), redox reaction  (?) have been studied for hundreds of years. Owing to the development of organic methodology (?), hundreds of millions of reactions have been practised and more and more compounds have been produced. Nevertheless, the mechanism of organic reactions has not been completely understood and the practicality of a new organic reaction still mainly relies on human judge from expertise and the eventual exploratory synthesis verification.

Modeling the organic reactions through physical-level method, such as quantum mechanical modeling, is a traditional way to recognize them, whereas it may lead to over-complicated model with poor informative representation  (?), even for simple reaction containing only several atoms, is essentially difficult to model due to the need of considering the combinatorial component arrangement using quantum chemistry method. Predicting a complicated chemical reaction under a certain condition is even much more challenging (?), because it requires considering every transition-state, the combination between molecules and their given environment.

Instead, the latest symbol model for chemical reaction has been proposed, in which chemical elements and molecules are regarded as symbols and reactions are considered as text with chemical information. Consequently, most text processing methods including machine learning, especially deep learning, can be applied to the text in chemical language. Support Vector Machine (SVM) has been proved to be useful to predict the result of crystallization of templated vanadium selenites  (?). However, it requires complicated manual feature selection with a basis of necessary chemical knowledge. Information retrieval is also an effective way to predict the products of organic reactions  (?, ?), which presents a limited candidate set for ranking. Continuous representation of molecules  (?) provides a convenient method to automatically generating chemical structures. More recently, some researchers  (?) cast the reaction prediction task as a translation problem by introducing a template-free sequence-to-sequence model, trained end-to-end and fully data-driven and achieved an accuracy of 80.1% without relying on auxiliary knowledge such as reaction rules. Recently, Abigail Doyle et al.  (?) proposed the random forest algorithm, which can accurately predict the yield of Buchwald Hartwig cross-coupling reactions with many detailed features of materials in reactions, though their computational model can only process a kind of reaction and needs too much information about the reactions.

Existing work using machine learning for chemical information processing falls either relying on strong chemical knowledge source or focusing on specific types of reactions. Distinctive from previous studies, we provide a cutting-edge symbol alone model on chemical text of organic reactions from a general background. A complete data-driven method is proposed for open type of chemical organic reactions, releasing the inconvenient prerequisite with chemical prior knowledge. Without complex parameter setting or manual chemical knowledge based feature selection, our approach can automatically discover the salient features and reaction patterns for effective reaction practicality judgment.

In recent years, natural language processing has popularly adopted embedding representation for text units which is a sort of low-dimensional continuous representation learned from neural networks. Following the latest advance of deep language processing, embedding is also used to represent chemical text segments for chemical reaction learning. Using a data-driven mode, our model will directly learn from a large scale of available reaction data. We use reaction formulation collected from the publications for about 1.7 million reactions. Practicality judgment can be straightforwardly formulized into a discriminative machine learning task over two types of reactions, positive and negative. However, the latter, negative reactions, are seldom reported in chemical literatures and thus usually hard collected. When quite a lot of positive reactions are collocated with few negative ones, the machine learning models have to struggle on seriously imbalanced training dataset. In this work, we propose an effective chemical rule based method for negative reaction generation to cope such a long-term big challenge. Eventually, given the reactants and products, our model can accurately judge the reaction practicality. Our model pipeline is given as follows. We first preprocess the SMILES sequence of each reactant and product in atom-wise and adopt an unsupervised segmentation algorithm to tokenize the resulting text into segments in a natural language processing way. Then the text or symbol difference between reactants and products which stands for the reaction steps is extracted and tagged from an edit distance detection operation on both sides of reaction text. For an effective representation, all the resulting chemical text segments are presented in an embedding form so that either the reactants or the products can be put into vector representation as well. At last, the reactants and the products which are all in vectors are fed to a neural network for practicality learning and judgment.

METHOD

For chemical reaction prediction, the key point is to effectively capture the internal relationships between a reactant and the corresponding product representation along with Reaction Symbol Distance (RSD). Note that we assume unreactive reactions will be always kept unreactive under all possible, known reaction conditions, thus we remove all reaction conditions in our judgment. This task is formulized into a language processing over the corresponding chemical text. As shown in Figure 1, the text segments of reactants and products are represented as vectors of low-dimensional embedding representation. Then, a deep neural network is trained to learn the chemical principles of reactions by transforming the feature representation of reactants and products. After training, given a reaction text input, the model will judge the practicality.

Unsupervised Tokenization

SMILES (Simplified Molecular Input Line Entry System) is a line notation for entering and representing molecules and reactions using short ASCII strings, which was initiated by David Weininger at the USEPA Mid-Continent Ecology Division Laboratory in Duluth in the 1980s (?). The primary reason SMILES is more useful than an extended connection table is that it is a linguistic construct, rather than a computer data structure. SMILES is a true language, albeit with a simple vocabulary (atom and bond symbols) and only a few grammar rules. SMILES representations of structure can in turn be used as “words” in the vocabulary of other languages designed for storage of chemical information and chemical intelligence. Some examples are shown in Table 1.

At the very beginning, we remove the hydrogen atoms and the atom mappings from the reaction string, and canonicalized the molecules. We treat chemical reaction described by SMILES as a kind of text in natural language. Considering that chemical elements (atoms) and various SIMILES bond symbols are “characters” in the chemical language, a sequence of SIMILES which stands for chemical compound can be regarded as the corresponding sentence. Therefore we need to mine the sequence to find a basic meaningful linguistic unit, word. As SMILES encoding text does not provide a word segmentation with solid chemical meaning to facilitate the chemical text processing, we turn to unsupervised tokenization solution in the existing natural language processing  (?). Therefore, we adopt goodness measure based method to tokenize each reactant, product text in SMILES into a sequence of “words”. Let $W=\{\{w_{i},g(w_{i})\}_{i=1,...,n}\}$ be a list of character $n$-grams (namely, word candidates) each associated with a goodness score for how likely it is to be a true word from a linguistic/chemical perspective, where $w_{i}$ is a word candidate and $g(w_{i})$ is its goodness function.

The adopted segmentation algorithm is a greedy maximal-matching one with respect to a goodness score.

[TABLE]

It works on $T$ to output the best current word $w^{*}$ repeatedly with $T=t^{*}$ for the next round as follows, with each $\{w,g(w)\}\in W$.

In our work, we use Description Length Gain (DLG) as the goodness measurement for a candidate character $n$-gram from the chemical text. In principle, the higher goodness score for a candidate, the more likely it is to be a true word. DLG was proposed by Kit and Wilks  (?) for compression-based unsupervised segmentation. The DLG extracts all occurrences of $x_{i..j}$ from a corpus $X=x_{1}x_{2}...x_{n}$ and its DLG goodness score is defined as

[TABLE]

where $X[r\to x_{i..j}]$ represents the resultant corpus by replacing all items of $x_{i..j}$ with a new symbol $r$ throughout $X$, and $\oplus$ denotes the concatenation operator. $L(\cdot)$ is the empirical description length of a corpus in bits that can be estimated by the Shannon-Fano code or Huffman code as below, following classic information theory (?),

[TABLE]

where $|\cdot|$ denotes the string length, $V$ is the character vocabulary of $X$ and $\hat{p}(x)$ is $x$’s frequency in $X$.

Reaction Symbol Distance (RSD) Generation

To formally represent the text difference from reactants to products in a reaction formula, we introduce the formal concept of Reaction Symbol Distance (RSD), which indicates how source chemical text can be transformed into target one through a series of symbol inserting and deleting operations. The text operation series can be decoded from calculating the edit distance  (?). Edit distance is used to quantify how dissimilar two strings are to one another by counting the minimum number of operations required to transform one string into the other.

For a source sequence $S=s_{1}s_{2}\ldots s_{n}$ and the target sequence $T=t_{1}t_{2}\ldots t_{m}$, the RSD sequence $R=r_{1}r_{2}\ldots r_{n}$ is encoded by the following tags:

• •

AD indicates a string should be added right before the corresponding location.

• •

AR indicates the corresponding symbol should be replaced by the given string with the tag.

• •

RR the corresponding symbol should be deleted.

• •

_ means that there is no operation at the location.

All compound sequences $S$ and $T$ are split into elements, and the resulting RSD from $S$ to $T$ are illustrated in the Figure 2.

The data processing steps together with examples are summarized in Table 2. The same preprocessing steps were applied to all datasets.

Embedding

In our adopted neural model, an embedding layer is used to map each element or segmented “word” from a sequence into a vector with dimension $d$. Our model takes three types of inputs, reactant, RSD and product. After embedding, the reactant sequence with $n$ words is represented as $\mathbb{R}^{d\times n}$. Similarly, we obtain the embeddings of the reactant sequence $\mathbf{R}$, the RSD sequence $\mathbf{S}$ and the product sequence $\mathbf{P}$. Then, the input sequences are subsequently aggregated into two compact representations through projection and concatenating:

[TABLE]

Siamese Network

In order to learn the optimal representations of chemical reactants $\mathbf{M_{1}}$ and products $\mathbf{M_{2}}$ with RSD, we propose to use a pair-based network structure called Siamese network which has been proven as an effective framework for image matching  (?), sequence similarity comparison tasks  (?, ?). Since the negative reaction instances are extremely insufficient and most reported yields concentrate in a narrow range, common neural network suffers from the imbalance learning difficulty. The structure of Siamese network consists of two identical branches that share weights and parameters. Each branch poses a deep neural network for feature learning. In this work, we adopt Long-Short Term Memory (LSTM) Network  (?) as the branch component due to its advance for sequence modeling. Figure 3 shows an LSTM based branch architecture. The LSTM unit is defined as follows.

[TABLE]

where $\sigma$ stands for the sigmoid function and the $\odot$ represents element-wiselayer to form a final representation. multiplication. $\oplus$ denotes vector concatenation. $\mathbf{i_{t}},\mathbf{f_{t}},\mathbf{u_{t}},\mathbf{c_{t}},\mathbf{h_{t}}$ are the input gates, forget gates, memory cells, output gates and the current states, respectively. Given a sequence input, the network computes the hidden sequence $\mathbf{h}_{t}$ by applying the formulation for each time step.

After embedding, the vectorized inputs $M_{1}$ and $M_{2}$ are separately fed to forward LSTM and backward LSTM (BiLSTM) to obtain the internal features of two directions. The output for each input is the concatenation of the two vectors from both directions: $\mathbf{h}_{t}=\overrightarrow{\mathbf{h}_{t}}\parallel\overleftarrow{\mathbf{h}_{t}}$. Hence, we have the processed representations of the reactant and product with RSD, $\hat{M_{1}}=BiLSTM(M_{1})$ and $\hat{M_{2}}=BiLSTM(M_{2})$. Then, our model concatenates the representation of $\hat{M_{1}}$ and $\hat{M_{2}}$ to a Multi-Layer Perception (MLP) layer to form a final representation. The output of the model is activated by a sigmoid function to ensure the prediction is in [0,1].

[TABLE]

where $\mathbf{x}$ is the output of MLP and $\mathbf{y}$ is the prediction.

Training objectives

For practicality judgment, we use binary cross entropy as the loss function.

[TABLE]

where $\hat{\mathbf{y}}_{t}$ denotes the prediction, $\mathbf{y}_{t}$ is the target and $t$ denotes the data index.

DATA

The reaction data for our model evaluation has 5 sources, (1) a public chemical reaction dataset USPTO, (2) a large scale of reaction dataset extracted from reports of Chemical Journals with High Impact factors (CJHIF), (3) ruled generated negative chemical reactions by Chemical.AI laboratory111http://www.chemical.ai, (4) real failed reactions from Chemical.AI partner laboratories and (5) real reactions from Chemical.AI laboratories.

Statistics

• •

**Positive reactions from USPTO (USPTO)

This public chemical reaction dataset was extracted from the US patents grants and applications dating from 1976 to September 2016 (?) by Daniel M. Lowe (?). The portion of granted patents contains 1,808,938 reactions described using SMILES 222https://figshare.com/articles/Chemical_reactions_from_US_patents_1976-Sep2016_/5104873. Such reaction strings are composed of three groups of molecules: the reactants, the reagents, and the products, which are separated by a ‘$>$’ sign. After data cleaning with RDKit (?**), an open-source cheminformatics and machine learning tool, it remained 269,132 items at last.

• •

Positive reactions from CJHIF(CJHIF)

3,219,165 reactions mined from high impact factor journals333The journal list is attached in the Supplemental Material. with reagent, solvent and catalyst information, in addition with yield. After data cleaning and selection, we used remaining 1,763,731 items at last.

• •

Rule-generated negative reactions from Chemical.AI (Chemical.AI-Rule)

For every product in the positive reaction sets, we adopt a set of chemical rules to generate all possible reactions which may output the respective products. Then we filter the resulted reactions by a very large known positive reaction set from Chemical.AI (which contains 20 million known reactions collected from chemical literatures and patents). Namely, all the remained unreported reactions are taken as negative reactions. Due to memory limitation, we keep 100K rule-generated negative reactions in our dataset.

Our idea for auto-construction of negative chemical reaction samples is actually quite intuitive, that is, we simply regard no-show reactions from any known literature are quite possibly negative ones. Only if the reference positive reaction set is large enough, we can receive quite reliable negative reasons from such filtering.

• •

Real negative reactions from Chemical.AI (Chemical.AI-Real-1)

12,225 real failed reactions from chemical experiment record of Chemical.AI partner laboratories. After data deduplication and canonicalization, it remained 8,797 reactions.

• •

Real reactions from Chemical.AI (Chemical.AI-Real-2)

24,514 real reactions from chemical experiment record of Chemical.AI partner laboratories, in which there are 16,137 positive reactions and 8,377 negative reactions, where the productivity of negative reactions is 0%. This data set is equally split into two parts: training set and test set.

Setup

For practicality judgment, we let the two positive sets collocate with the two negative dataset to form four combinations. The distributions of positive and negative reaction from the train/dev/test sets are in Table 3.

In our experiments, the ratio of the training set and the test set is 9:1 and 10% of the training set is held out as development (dev) set444Dev set is used to supervise the training process in case of over-fitting or under-fitting in deep learning scenerio.. For practicality judgment, since there are no negative samples from USPTO dataset, we use the positive reactions from CJHIF and USPTO to pair the real negative and rule-generated negative samples from Chemical.AI. For the data collected from laboratories (Chemical.AI-Real-2), we take them as the test set to examine the generalization ability of our model.

Considering the calculation efficiency, we specify a max length of 100 words for each SMILES sequence and apply truncating or zero-padding when needed. The embedding weights are randomly initialized with the uniformed distribution in the interval [-0.05, 0.05].

Evaluation metrics

Our practicality judgment evaluation is based on the following metrics: Accuracy, Precision, Recall and F1-score. Four types of predictions are as shown in Table 4.

Accordingly, we can calculate the performance of Accuracy, Precision, Recall and F1-score as follows.

[TABLE]

EXPERIMENT

Practicality Judgment

Given input sequences, the model will output the reaction success probabilities. To evaluate the result, a threshold is required to distinguish from positive or negative predictions. According to our preliminary experiments, the threshold is set to 0.5555This is also the common setting for binary classification tasks and our quantitive study shown in Figure 5 also verifies the optimal setting.. The experimental result is shown in Table 8. We observe the positive reactions could be recognized essentially (nearly 99%). Though the proportion of positive and negative cases is over 30:1, our model also ensures a high negative F1-score more than 72%. Besides the rule-based negative dataset have a higher Negative-F1-score. From the statistics of the datasets, we know the rule-based negative dataset is much bigger than the Chemical.AI-Real-1, which not only alleviates the imbalance between positive and negative examples but also increases the diversity of negative examples.

Generalization Ability

In order to demonstrate the generation ability of our learning model, we report the judgment results on Chemical.AI-Real-2 dataset with different training settings in Table 9.

As the Chemical.AI-Real-2 comes from true laboratory record, our model prediction is actually evaluated in these real chemical experiments. Note that these negative reactions were expected to work by experienced chemists, which means they are literally correct in chemistry rules and the chemists must encounter difficulties to predict the practicality of these reactions. Therefore, when our model gives correct practicality predication over these actually failed reactions, it means that our model performs better than human experts in these cases.

As we know, it is difficult to get sufficient enough failed reactions because they are rarely reported in literature. Meanwhile negative examples are indispensable in discriminative machine learning on these two types of reactions. Here we show that rule-generated negative reactions for training set building may yield remarkable prediction accuracy in real chemical reaction records by considering that the rule negative samples perform best among all training settings in Table 9. This opens a new way in the research of chemistry reaction prediction.

To have an insight of how the thresholds affect the model performance on the Chemical.AI-Real-2 dataset, we record the predication results by ranging the thresholds from 0.1 to 1 with step=0.1. The visualization results are shown Figure 5, which shows the best performance when the threshold is 0.5.

Figure 6 illustrates the ROC (Receiver Operating Characteristic) curve which relates to the diagnostic ability of a binary classifier system, and the Area under the Curve (AUC) of ROC is 80.90%, which means this model can perform quite well when the threshold sets rightly.

Incremental Experiment

Different datasets may have different statistical distribution characteristics on reaction types. To fully examine the capacity of our model, we conduct a series of incremental experiments by mixing a small part of different dataset to the origin one and using the rest as test set.

We divided the Chemical.AI-Real-2 dataset into two parts, the incremental set and test set in the ratio of 1:1. Than we add different sized parts of the incremental set with ratios [0.1, 0.2, …, 0.9, 1.0] to the training set (USPTO + Chemical.AI-Real-1) and conduct the experiments, respectively.

The results in Figure 4 show that even there is only a small amount of data added into the training set as the same source as the test set, the judgment results will be improved greatly.

DLG Segmentation

The adopted unsupervised tokenization over the SMILES text is based on a set of “words” with significant DLG scores in terms of the goodness measure methods. Despite its usefulness in our computational process, we also observe their chemical meaning. Table 5 lists a small part of the “words” with high DLG scores. For example, it is not strange to any chemists that the structure of metal complex is the key to many organic reactions and in the words list, we find the number and the metallic element are always put in a same fragment which means that the ligands’ position information attaching to the metallic element is useful for a better and more accurate embedding representation in our model. At the meanwhile, most of ordinary functional groups are also in the same fragment, like C=C, C#C, C=O and C#N, which means the model regards them as a group to process the reaction like what organic chemists do in their research. We also find that the ring structure in a molecule is always divided in different fragments. Though seemingly irrational, the model actually recognizes different functional parts in a ring for more targetedly processing, which is indeed helpful to extract the reaction pattern in the later process.

Ablation Experiment

We investigate the effect of different features in our model by removing them one by one. As shown in Table 6 all the features contribute to the performance of our final system. If we remove either RSD or DLG Tokenizaton, the performance drops. This result indicates that both features play an important and complementary role in the feature representation.

Recurrent Neural Network Types

We also compare Siamese network with the different standard recurrent neural networks - LSTM, BiLSTM, GRU and BiGRU, and the comparison of the results is demonstrated in Table 7. Obviously, Siamese network outperforms all the others, especially on the negative cases, which shows Siamese network could effectively handle the data imbalance issue.

Significance in Chemistry

In practice, we have shown that our model uses the rule-based generated data as negative examples to train the model, while in the test, our model obtains significant judgment accuracies in real reaction record. It means that our model has been capable of extracting the feature of both positive and negative examples and filtering the bias introduced by rule, which shows remarkable modeling ability and will helps to improve the development of chemical engineering. During the test, we find that model can recognize some reactions which seems to be impractical but can react actually and some reactions which seems to be reactive but cannot react actually. Figure 7 and Figure 8 show such highly confused examples.

CONCLUSION

We present a deep learning model to model real-world chemical reactions and unearth the factors governing reaction outcomes only from symbol representation of chemical information. In contrast to conventional methods which require massive manual features or are only evaluated on small datasets for specific reaction types, our approach is much more simple, end-to-end and effective. Especially, we use a rule-based method to generate unlimited negative samples, and the results evaluated on real reaction records show satisfactory judgment performance. In a distinctive perspective, this work reveals the great potential to employ deep learning method to help chemists judge the practicality of chemical reactions and develop more efficient experimental strategies to reduce the cost of invalid experiments. The resultant model can be used more than practicality judgment, but has a potential to help effective synthesis route design which has been an ongoing task in our current study and chemical practice.666Our prediction models have been online, the link is http://bcmi.sjtu.edu.cn/~dl4chem

References and Notes

G. Casiraghi, L. Battistini, C. Curti, G. Rassu, F. Zanardi, Chemical Reviews 42, 3076 (2011).

• 2.

Z. Abu El-Rub, E. Bramer, G. Brem, Industrial Engineering Chemistry Research 43, 6911 (2004).

• 3.

S. Caron, N. M. Do, J. E. Sieser, D. C. Whritenour, P. D. Hill, Organic Process Research and development 2, 324 (2009).

• 4.

H. Yao, et al., ACS Applied Materials and Interfaces 6, 3575 (2016).

• 5.

G. C. Fu, ACS central science 3, 692 (2017).

• 6.

O. Wiest, D. C. Montiel, K. N. Houk, The Journal of Physical Chemical 101, 8378 (1997).

• 7.

Z. L. Song, C. A. Fan, Y. Q. Tu, Chemical Review 111, 7523 (2011).

• 8.

J. A. Rincón, et al., Organic Process Research and development 15, 1428 (2011).

• 9.

Z. Flisak, W. Sun, ACS Catalysis 5, 4713 (2015).

• 10.

S. E. Denmark, The Journal of Organic Chemistry 74, 2915 (2009).

• 11.

A. Streitwieser, The Journal of Organic Chemistry 74, 4433 (2009).

• 12.

R. E. Plata, D. A. Singleton, Journal of The Americal Chemical Society 137, 3811 (2015).

• 13.

P. Raccuglia, et al., Nature 533, 73 (2016).

• 14.

C. W. Coley, R. Barzilay, T. S. Jaakkola, W. H. Green, K. F. Jensen, ACS Central Science 3, 434 (2017).

• 15.

J. N. Wei, D. Duvenaud, A. Aspuruguzik, ACS Central Science 2, 725 (2016).

• 16.

R. Gómez-Bombarelli, et al., CoRR abs/1610.02415 (2016).

• 17.

P. Schwaller, T. Gaudin, D. Lanyi, C. Bekas, T. Laino, arXiv preprint arXiv:1711.04810 (2017).

• 18.

D. T. Ahneman, J. G. Estrada, S. Lin, S. D. Dreher, A. G. Doyle, Science (2018).

• 19.

D. Weininger, Journal of Chemical Information and Computer Sciences 28, 31 (1988).

• 20.

H. Zhao, C. Kit, Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I (2008).

• 21.

C. Kit, Y. Wilks, Proceedings of 1998 International Conference on Chinese Information Processing (1998), pp. 223–229.

• 22.

C. E. Shannon, A Mathematical Theory of Communication, vol. 27 (Blackwell Publishing Ltd, 1948).

• 23.

S. B. Needleman, C. D. Wunsch, Journal of molecular biology 48, 443 (1970).

• 24.

I. Melekhov, J. Kannala, E. Rahtu, International Conference on Pattern Recognition (2016), pp. 378–383.

• 25.

J. Mueller, A. Thyagarajan, Thirtieth AAAI Conference on Artificial Intelligence (2016), pp. 2786–2792.

• 26.

S. Chopra, R. Hadsell, Y. Lecun, Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on (2005), pp. 539–546 vol. 1.

• 27.

S. Hochreiter, J. Schmidhuber, Long short-term memory (Springer Berlin Heidelberg, 1997).

• 28.

D. Lowe, Chemical reactions from us patents (1976-sep2016) (2017).

• 29.

D. M. Lowe, Extraction of chemical structures and reactions from the literature, Ph.D. thesis, University of Cambridge (2012).

• 30.

G. Landrum, et al., rdkit/rdkit: 2017_09_1 (q3 2017) release (2017).