The Materials Science Procedural Text Corpus: Annotating Materials Synthesis Procedures with Shallow Semantic Structures

TL;DR

This paper introduces a new annotated dataset of materials synthesis procedures in scientific text, enabling better extraction of structured information for scientific analysis and automation.

Contribution

The authors present a novel, expert-annotated corpus of materials synthesis procedures with semantic graphs, facilitating research in automated information extraction from scientific literature.

Findings

Created a dataset of 230 annotated synthesis procedures

Highlighted challenges in annotating scientific text with semantic structures

Made the corpus publicly available for research use

Abstract

Materials science literature contains millions of materials synthesis procedures described in unstructured natural language text. Large-scale analysis of these synthesis procedures would facilitate deeper scientific understanding of materials synthesis and enable automated synthesis planning. Such analysis requires extracting structured representations of synthesis procedures from the raw text as a first step. To facilitate the training and evaluation of synthesis extraction models, we introduce a dataset of 230 synthesis procedures annotated by domain experts with labeled graphs that express the semantics of the synthesis sentences. The nodes in this graph are synthesis operations and their typed arguments, and labeled edges specify relations between the nodes. We describe this new resource in detail and highlight some specific challenges to annotating scientific text with shallow semantic structure. We make the corpus available to the community to promote further research and development of scientific information extraction systems.

Tables

Table 1. (a)

Entity type	Count
Material	4843
Number	4095
Operation	3786
Amount-Unit	1659
Condition-Unit	1621
Material-Descriptor	1430
Condition-Misc	535
Synthesis-Apparatus	490
Nonrecipe-Material	475
Brand	348

Table 2. (a)

Entity type	Count
Material	4843
Number	4095
Operation	3786
Amount-Unit	1659
Condition-Unit	1621
Material-Descriptor	1430
Condition-Misc	535
Synthesis-Apparatus	490
Nonrecipe-Material	475
Brand	348

Table 3. (b)

	Recipe-target, Solvent-material,
Operation-argument	Atmospheric-material, Recipe-precursor,
relations	Participant-material, Apparatus-of
	Condition-of
Non-operation entity	Descriptor-of, Number-of, Amount-of,
relations	Apparatus-attr-of, Brand-of, Coref-of,
Operation-Operation	Next-operation
relations

Table 4. Table 2: Various dataset statistics. Additional details provided in referred figures. To determine unique operations, Operation entity mentions are lemmatized with the WordNet lemmatizer and the unique lemmas are counted.

Item	Count
Synthesis procedures	230
Sentences	2113
Tokens	56510
Entity mentions	20849
Entities	4883
Unique operations	409
Entity types (Table 1a)	21
Relation types (Table 1b)	14
Avg. sentence length (Fig. 3b)	26
Avg. sentences/synthesis procedure (Fig. 3a)	9

Table 5. (a)

Entity type	Fleiss’ Kappa
Material	0.916
Number	0.971
Operation	0.859
Amount-Unit	0.967
Condition-Unit	0.985
Material-Descriptor	0.638
Condition-Misc	0.784
Synthesis-Apparatus	0.860
Nonrecipe-Material	0.371
Brand	0.862

Table 6. (a)

Entity type	Fleiss’ Kappa
Material	0.916
Number	0.971
Operation	0.859
Amount-Unit	0.967
Condition-Unit	0.985
Material-Descriptor	0.638
Condition-Misc	0.784
Synthesis-Apparatus	0.860
Nonrecipe-Material	0.371
Brand	0.862

Table 7. (b)

Annotation	Fleiss’ Kappa
Span-level labels	0.861
Relation labels	0.941

Full text

1 Introduction

††footnotetext: ∗Equal contribution

Systematically reducing the time and effort required to synthesize novel materials remains one of the grand challenges in materials science. Massive knowledge bases which tabulate known chemical reactions for organic chemistry Lawson et al. (2014) have accelerated data-driven synthesis planning and related analyses Segler et al. (2018); Coley et al. (2017). Automated synthesis planning for organic molecules has recently achieved human-level planning performance using massive organic reaction knowledge bases as training data Segler et al. (2018). There are, however, currently no comprehensive knowledge bases which systematically document the methods by which inorganic materials are synthesized Kim et al. (2017a, b). Despite efforts to standardize the reporting of chemical and materials science data Murray-Rust and Rzepa (1999), inorganic materials synthesis procedures continue to reside as natural language descriptions in the text of journal articles. Figure 1 presents an example of such a synthesis procedure. To achieve similar success for inorganic synthesis as has been achieved for organic materials, we must develop new techniques for automatically extracting structured representations of materials synthesis procedures from the unstructured narrative in scientific papers Kim et al. (2017b).

To facilitate the development and evaluation of machine learning models for automatic extraction of materials syntheses from text, in this work we present a new dataset of synthesis procedures annotated with semantic structure by domain experts in materials science. We annotate each step in a synthesis with a structured frame-semantic representation, with all the steps in a synthesis making up a Directed Acyclic Graph (DAG). The types of nodes in the graph include synthesis operations (i.e. predicates), and the materials, conditions, apparatus and other entities participating in each synthesis step. Labeled edges represent relationships between entities, for example Condition_of or Next_Operation. Our dataset consists of 230 synthesis procedures annotated with these structures. An example sentence level annotation is given in Fig. 2. We make the corpus available to the community to promote further research and development of scientific information extraction systems for procedural text.111Public dataset: https://bit.ly/2WLCbyh

2 Description of the Annotated Dataset

Here we describe the manner in which synthesis procedures were chosen for annotation (§2.1), present a description of the structures we annotate (§2.2), summarize key statistics of the dataset (§2.3), highlight specific annotation decisions (§2.4) and present inter-annotator agreements (§2.5). All annotations were performed by three materials scientists using the BRAT222http://brat.nlplab.org/ annotation tool (Stenetorp et al., 2012).

2.1 Selecting Synthesis Procedures for Annotation

The 230 synthesis procedures annotated were selected from our database of 2.5 million publications describing materials synthesis. The database was built from agreements with major scientific publication companies. Synthesis procedure text were obtained by first parsing the HTML text of the full publications, then automatically identifying candidate synthesis paragraphs using a trained classifier. This paragraph classifier was trained on a set of manually labeled paragraph examples and has a F1 score of 90.2 on a held out test set.333The labeled data for this classifier is not part of the data release associated with this paper due to licensing restrictions from publishers. The paragraphs selected by the classifier were manually verified as containing complete, valid materials synthesis procedures by domain experts. While a given synthesis procedure is most often a single paragraph, there are cases where it spans multiple paragraphs, we consider all the paragraphs to be a single synthesis procedure. All the semantic structures were then manually annotated in these selected synthesis procedures. Fig 1 depicts an example paragraph containing a synthesis procedure. In selecting a synthesis procedure for annotation, a small number of valid synthesis procedures ($\sim 20\%$) are ignored; this is done for the synthesis procedures which are not amenable to annotation from a sentence-level frame-semantic view of synthesis steps, or ones in which most entities in the synthesis do not agree with our definitions of operations and argument entities (see §2.4 for further discussion).

2.2 Structures Annotated

An annotated graph consists of nodes denoting the participants of synthesis steps and edges denoting relationships between the participants in the synthesis. Operation nodes define the main structure of the graph and the arguments for each operation include different materials, conditions and apparatus. For annotating the text describing a synthesis procedure, we define a set of span-level labels that identify the operations and typed arguments in the text, i.e. the nodes of the graph. We also define a set of relationships between text spans, which label the edges of the synthesis graph. We detail these two kinds of labels next.

Span-level Labels: Each span is a sequence of tokens or characters which form one entity mention (for example. “quartz tube furnace”). Entity mentions are associated with entity types which specify a category/kind for the entity mention. Our dataset defines a total of 21 entity types, with the least frequent entity type occuring 32 times. The 10 most frequent entity types defined for our dataset are listed in Table 1a. We describe a notable subset of the entity types in more detail below alongside examples of their occurrence in text. In examples, the text underlined is the span to be annotated.

Material: Materials that are used in the synthesis of the target. For example: \chCr_2O_3, Strontium carbonate, \chBaTiO_3, \chLi_2CO_3, Water, Ethanol.

Nonrecipe-Material: Chemically specified materials that are not used in the synthesis of the synthesis target. For example: “\chBaTiO_3 powder (\chBa/\chTi=0.999)”, “\chLi2CO3 was used as the Li source”, “Si/Al ratio was 5”.

Operation: Discrete actions physically performed by the researcher or discrete process steps taken to synthesize the target.

Material-Descriptor: Describes a material’s structure, shape, form, type, role, etc. and must be directly or nearly adjacent to the material it describes. Does not include amounts, concentrations, or purities of materials. For example: \chCaCu_3Ti_4O_12 compound, Copper ion, \chGaAs nanowires, Anatase \chTiO_2, Deionized water.

Meta: A canonical name to specify a particular overall synthesis method used for synthesis. For example: “Graphite oxide was prepared by oxidation of graphite powder according to the modified Hummers’ method”. “Bi2S3 nanorods with orthorhombic structure were prepared through the hydrothermal method”. “Graphene oxide (GO) was prepared from graphite powder by the Staudenmaier method.”

Amount-Unit: These describe absolute amounts, concentrations, purities, ratios, flow rates and so on. For example: mg, mL, M, %.

Condition-Unit: These describe the units of measurement for intangible conditions under which operations are performed. For ex: °C, K, sec, RPM, mW.

Condition-Misc: Qualitative descriptions of conditions. For example: Room temperature, Dropwise, Naturally, Vacuum.

Synthesis-Apparatus: Equipment used to perform an operation involved in the synthesis.

Characterization-Apparatus: Equipment used to characterize a material’s properties.

Relation Labels: We define a set of relationships between entity mentions, which label the edges of the synthesis graph. A subset of these relations describe direct relationships between operations and their arguments, others describe relationships between argument mentions, and the Next-Operation relation describes relationships between operations so as to step towards annotating full recipe graphs. The different relation labels we define are tabulated in Table 1b, a subset of these are defined below:

Recipe-target: Indicates a material assigned to an operation which is the target of the synthesis procedure.

Participant-material: A material that is part of a particular synthesis step.

Recipe-precursor: Indicates a material which is the source of an element for the target material used in a specific synthesis operation.

Apparatus-of: Denotes an apparatus to be used in a synthesis operation.

Condition-of: Denotes a reaction condition for a synthesis operation.

Coref-of: Intended to capture coreferent mentions of entities presented by abbreviations, text in parenthesis and so on. For example: “Air (\chO_2/\chN_2 mixture gases)” and “He were supplied to the porous support side …”. “Air” is coreferent with \chO_2 and \chN_2.

Amount-of: Links a number entity to the corresponding unit of measurement.

Next-operation: A relation intended to denote the true synthesis order of the synthesis steps; the relation is also intended to implicitly denote the flow of intermediate materials in the synthesis. However, in this first release of the data, as a placeholder for future annotations, Next-Operation is used simply used to indicate the next operation in text order rather than in true synthesis order.

We refer readers to our annotation guidelines for definitions of the complete set of entity type and relation labels in the dataset.

2.3 Dataset Statistics

Some key statistics of the dataset such as number of documents, tokens, entities and unique operations are listed in Table 2 and Fig. 3.

In reporting these statistics we perform tokenization and sentence segmentation using the ChemDataExtractor package Swain and Cole (2016).444https://pypi.org/project/ChemDataExtractor/1.2.2/

2.4 Annotation Decisions

Next we highlight specific points of contention in creating the current set of annotations.

What constitutes an operation?: While our definition of the Operation entity type specifies only actions performed by a lab researcher to be valid operations, there are cases where our annotations diverge from this definition. This happens in the following cases:

• •

Cases where an operation isn’t explicitly performed by the researcher. For example: “After this, the autoclave was cooled to room temperature naturally”.

• •

Cases with nested verb structures. For example: “white precipitate which was harvested by centrifugation …”.

In the current set of annotations, we allow experts to decide when a particular candidate operation should be considered valid and when it can be omitted. As our inter-annotator agreements will demonstrate, experts tend to agree often on what should be considered an operation. The question of what constitutes an operation is analogous to the notion of what constitutes an “event” in the broader NLP literature as highlighted by Mostafazadeh et al. (2016).

Argument state and argument re-use: Annotation of semantic structures often allow for argument spans to have multiple parents Surdeanu et al. (2008); Banarescu et al. (2013); Oepen et al. (2015). For example in Figure 2, the material “Cu” could be considered an argument of the operations “placed” and “heated”. Allowing for arguments to have multiple parents however runs into complications when the operation causes the state of a material to change (incidentally, this is not the case in the example we highlight above). When a materials state changes due to a specific operation, considering the same text span to be the argument of a different operation would not be chemically valid. For example, in the sentence:

After that, the mixed solution was aged at 60 degC for 48 h, followed by heating at 900 degC for 2 h with a heating rate of 5 degC min-1 in an N2 atmosphere.

“solution” is labeled as Participant-material for “aged”, but it isnt considered a Participant-material to “heating” since aging caused it to be a different material. Similarly, in:

1.6632 g lithium acetate was dissolved into 26 mL of ethanol-water mixture (12:1 in volume) and slowly dropped into the above suspension.

“lithium acetate” is only labeled as Participant-material for “dissolved” and not for “dropped” whose sole argument is “suspension”. This clearly highlights an instance of a material entirely absent from the text being the true argument of an operation. Therefore the current set of annotations does not allow for arguments to have multiple parents. Further, the tracking of state itself is also complicated by the difficulty in being able to write down precise states at a meaningful level of granularity for all possible materials, this is further complicated by the ambiguity presented by under-specified materials in synthesis text, for example in the sentence:

With the indraught of ozone, black solid appeared gradually and the clear solution turned into black slurry finally.

Most of the entities, “black solid”, “clear solution” and “black slurry” are chemically under-specified, with precise specification even unnecessary for describing the synthesis procedure.

Relations across sentences: Often, in synthesis procedures, a given synthesis step is described across multiple sentences. In these cases it would be meaningful to allow for relationships between operation-argument entities which are in different sentences. For the sake of simplicity and to stick more closely to a sentence level shallow semantic annotation, our current iteration of the annotations has avoided this annotation, however a very small number of instances of cross-sentence relations do exist ($<1\%$ of all relations in the dataset). Examples of this type are as follows:

First, sulfuric acid and nitric acid were mixed well by stirring 15 min in an ice bath, and then graphite powder was dispersed into the solution. After 15 min, potassium chlorate was added into the system - very slowly to prevent strong reaction during the oxidation process.

Oxygen with 20 sccm flow rate and argon with 40 sccm flow rate were used as the sputtering gas. Growth temperature was 400 degC and the RF power was 90 W.

Here “min” and “dispersed” are related by a Condition_Of relation. Similarly, “degC” and “W”, both are annotated with Condition_Of relations to “used”. Annotations of this kind were created when annotators deemed such an annotation absolutely necessary. Synthesis procedures which required annotation primarily of cross-sentence relations were ignored.

2.5 Inter-annotator Agreement

Next we report a host of inter-annotator agreements for the different levels of semantic annotation in our dataset. The agreements we report are based on a collection of 5 synthesis procedures which were annotated separately by all three expert annotators. All the numbers we report are Fleiss’ Kappa scores for the 3 expert annotators.

Span-level Labels: Agreements on span level labels correspond to the agreement on entity type labels assigned to individual tokens. We observe the overall agreement on the token level labels to be 0.861. A break down of this agreement by the entity type is presented in Table 3a. As this indicates, there seems to be high agreement on labels which have clear definitions; namely. Number, Amount-Unit. Labels which by definition are a lot more ambiguous, however, have a lower agreement. The two entity types Material-Descriptor and Nonrecipe-Material see the lowest agreements. We believe these to be inherently more subjective entity types. In the case of Material-Descriptor it is often that some annotators may consider the descriptor and the adjacent material to be Material in its own right, for example: “Deionized Water” may be considered as a material in its own right or ”deionized” may be considered to be a descriptor. In the case of Nonrecipe-Material, a similarly harder decision needs to be made by the annotator, since these are materials which aren’t involved in the synthesis but are still mentioned in the text for completeness information. Often it is up to the interpretation of the annotator to decide whether a material is indeed involved in the synthesis leading to the low agreement on this entity type.

Relation Labels: Agreement on relation labels were computed for the set of cases where a pair of annotators agreed on the token level annotations, this happens 66% of the time in our repeated annotations. For a pair of entities, if both annotators indicate the same relation type the annotators are considered to be in agreement. For relation labels we observe a agreement score of 0.941. Since we only consider cases where the token labels are in agreement, we believe that it is likely that when annotators agree on the token level annotations they also tend to agree on the relation level labels.

3 Related Work

Shallow semantic parsing in NLP: Prior work in the NLP community has defined and annotated semantic structures for text. These structured representations often seek to generalize about sentence level predicate-argument structure, abstracting away from the surface nuances of natural language and representing its semantics Abend and Rappoport (2017). A large body of work has created these resources for non-scientific text, as done in PropBank Palmer et al. (2005); Surdeanu et al. (2008), FrameNet Fillmore and Baker (2010), AMR Banarescu et al. (2013), semantic dependencies Oepen et al. (2015) and ACE event schemas Doddington et al. (2004). The GENIA project has defined event structures for biomedical data Kim et al. (2003) while Garg et al. (2016) extended the AMR framework to biomedical text. Closer still to the work presented here, Mori et al. (2014) have annotated cooking recipes with sentence and discourse level semantic relations. There has also been an interest in labeling scientific wetlab protocol text, with semantic structures and to facilitate training supervised models for the extraction of these structures Kulkarni et al. (2018). Kulkarni et al. make use of an altered version of the EXACT2 ontology, created for the annotation of biomedical procedural text Soldatova et al. (2014). The dataset presented here can be viewed to fit within the theme of sentence level semantics for procedural text, specifically tailored to materials science synthesis.

Materials Science & Chemistry: Prior work in the materials science community have shown that manual extraction and subsequent text mining can be an effective approach to analysis of synthesis routes for specific compounds and classes of materials Raccuglia et al. (2016); Ghadbeigi et al. (2015); these approaches however have been limited by scale due to the manual extraction step. There has also been strong a consensus that comprehensively extracting the knowledge contained within written inorganic materials syntheses is a key step towards reducing the overall discovery and development time for novel materials Butler et al. (2018). We believe that the dataset we release fills an important gap in the existing work on extraction of inorganic materials synthesis procedures, by allowing exploration into extraction at a scale not attempted before. Parallel with this work, work by Kim et al. (2018) and Tamari et al. (2019) adopt the dataset released here to aid extraction of structured representations from synthesis procedures and with Kim et al. presenting early experiments in synthesis planning from extracted synthesis.

The focus of existing datasets and resources in the materials science community, has been on materials structures and properties knowledge bases Jain et al. (2013), rather than reactions and synthesis. In pursuit of more scalable methods for materials synthesis data extraction, Young et al. (2018) have made use of automated methods for extracting specific categories of materials synthesis parameters, while Mysore et al. (2017) and Kim et al. (2017a) have both presented preliminary methodological explorations for automated extraction of elements of a synthesis graph from materials science literature. However, these lines of work have not presented general purpose annotated data with which to train information extraction models for extraction of structured synthesis representations at scale, the focus of this work.

4 Conclusion and Future Directions

In this work we present a shallow semantic parsing dataset consisting of 230 synthesis procedures. The dataset was annotated by domain experts in materials science. We also highlight specific difficulties in the annotation process and present agreement metrics on the different levels of our annotation. We believe the dataset will enable the development of robust supervised entity tagging models and is suitable for evaluating models trained to extract shallow semantic structures. This is evidenced by the adoption of the dataset by work contemporaneous with this work Kim et al. (2018); Tamari et al. (2019).

Future work in the development of this dataset could involve methods for the scaling up of the annotation process, perhaps by adapting the guidelines to enable annotation by non-experts at some stages of the annotation process. Further, we also plan to quantitatively establish the limits of our annotation schema for the kinds of information it isn’t able to capture. We also plan to add additional layers of annotation, including: co-reference relations between synthesis steps, states of argument entities, and linking annotated entities to entries in materials science knowledge bases such as The Materials Project.555https://materialsproject.org/

5 Acknowledgements

The authors would like to acknowledge funding from the National Science Foundation Award 1534340/1534341 DMREF and support from the Office of Naval Research (ONR) under Contract No. N00014-16-1-2432. Early work was collaborative under the Dept. of Energy’s Basic Energy Science Program through the Materials Project under Grant No. EDCBEE.