Understanding Spatial Language in Radiology: Representation Framework, Annotation, and Spatial Relation Extraction from Chest X-ray Reports using Deep Learning

TL;DR

This paper introduces Rad-SpRL, a framework for extracting detailed spatial information from radiology reports using deep learning, achieving high accuracy in identifying spatial roles in chest X-ray interpretations.

Contribution

It presents a novel annotation scheme and a deep learning NLP method for extracting spatial roles from radiology reports, with significant performance results.

Findings

Achieved F1 scores of 90.28 and 94.61 for Trajector and Landmark roles.

Moderate performance for Diagnosis and Hedge roles with F1 scores around 71-73.

Annotated 2000 reports to create a new corpus for spatial information extraction.

Abstract

We define a representation framework for extracting spatial information from radiology reports (Rad-SpRL). We annotated a total of 2000 chest X-ray reports with 4 spatial roles corresponding to the common radiology entities. Our focus is on extracting detailed information of a radiologist's interpretation containing a radiographic finding, its anatomical location, corresponding probable diagnoses, as well as associated hedging terms. For this, we propose a deep learning-based natural language processing (NLP) method involving both word and character-level encodings. Specifically, we utilize a bidirectional long short-term memory (Bi-LSTM) conditional random field (CRF) model for extracting the spatial roles. The model achieved average F1 measures of 90.28 and 94.61 for extracting the Trajector and Landmark roles respectively whereas the performance was moderate for Diagnosis and Hedge roles with average F1 of 71.47 and 73.27 respectively. The corpus will soon be made available upon request.

Tables

Table 1. Table 1: Comparison of our corpus with the studies extracting clinically-relevant radiological entities from chest X-ray and chest CT reports.

Paper	Finding	Anatomy	Descriptor	Diagnosis	Device	Hedge	Negation	Relation
Hassanpour et al. [11]	✓	✓	✓			✓	✓
Cornegruta et al. [12]	✓	✓	✓		✓		✓
Bustos et al. [13]	✓	✓		✓
Hassanpour et al. [14]	✓
Irvin et al. [8]	✓			✓	✓		✓
Annarumma et al. [15]	✓	✓	✓
Wang et al. [7]	✓			✓
Rad-SpRL (this paper)	✓	✓	✓	✓		✓		✓

Table 2. Table 2: Studies focusing on spatial relations in radiology reports.

Paper	Finding	Anatomy	Diagnosis	Hedge
Roberts et al. [19]	—	✓ (Spatially related to a finding)	—	—
Rink et al. [18]	✓	✓ (Linked with finding)	—	✓ (Not linked with finding/location)

Table 3. Table 3: Studies who have used OpenI manual annotations.

Paper	How OpenI chest X-ray dataset is involved	Limitation (Radiology entities annotated/considered for model evaluation)
Demner-Fushman et al. [6]	Manually annotated or coded the collected 3996 reports with findings, diagnoses, body parts using MeSH terms supplemented by RadLex codes. Automatic annotation was also produced by the Medical Text Indexer (MTI).	This is a manual annotation process relying on MeSH terms and standard qualifier terms. The coded terms were not well-distinguished between findings and diagnoses. Moreover, the annotation lacks other information such as relation between findings and diagnoses. The automatic labeling does not include the related body parts for the labeled finding. (Positive Findings/Diagnoses and Body parts)
Shin et al. [24]	Trained CNNs using already available image annotations from Demner-Fushman et al. [6] and considered images where each image is labeled with only a single disease label using unique MeSH term combinations (this accounted for around 40 percent of the full OpenI dataset and 17 unique disease annotation patterns). Generated image annotations including disease as well as its contexts such as location, severity, and the affected organs by taking into account image/text contexts while training CNNs.	Although the annotation includes disease context, it only generates image caption corresponding to one disease per image. However, it is highly likely that multiple diseases or findings may be associated with an image. (Findings/Diagnoses and their context such as location and severity)
Wang et al. (2017) [7]	Used text mining (DNorm [26] and MetaMap [27]) to label disease names using reports. Evaluated their image labeling method on OpenI reports using the key findings/disease names coded by human annotators as gold standard [6].	Only used the available annotations for evaluating their proposed method. (Findings/Diagnoses)
Wang et al. (2018) [20]	Evaluated a text-image embedding auto-annotation framework on the OpenI dataset using the key findings/disease names coded by human annotators as the gold standard [6].	Used the annotated OpenI dataset for evaluating proposed disease classification method for 14 diseases. (Findings/Diagnoses)
Peng et al. [21]	Defined rules utilizing universal dependency graphs to identify negation or uncertainty related to findings. Manually checked the annotations in OpenI and organized the findings into 14 domain-important and generic types of medical findings.	Used the OpenI dataset both for designing the patterns and testing. Although they mentioned that organizing the findings into fine-grained categories can facilitate in correlating findings with the diagnosis, the terms distinguished as diagnoses or body parts were not utilized in the study for showing any correlation. (Findings)
Daniels et al. [22]	Proposed a deep neural network that predicts one or more diagnoses given an image by jointly learning visual features and topics from report findings.	Used the OpenI dataset and their corresponding ‘findings’ annotations both for fine-tuning and evaluating the model. (Findings/Diagnoses)
Huang et al. [25]	Proposed a neural sequence-to-sequence model by leveraging “indication” information of the report which includes annotating relationship between the positions where the finding term appears. They used the OpenI manual annotations as a reference annotation for evaluating the model.	Although this generated annotations for multiple diseases per image and also aimed to improve the results of Shin et al.[24] in annotating disease along with context such as location and severity, they did not annotate other useful contexts including spatial information of the finding as well as the associated diagnosis. (Findings/Diagnoses and their context such as location and severity)
Zech et al. [23]	To assess the generalizability of a deep learning model for screening pneumonia across 3 hospital systems. Used human-annotated pathology labels of OpenI dataset for testing.	Used OpenI only for evaluation. (Findings/Diagnoses)
Candemir et al. [28]	Fine-tuned several deep CNN architectures to detect presence of cardiomegaly. Used OpenI dataset both for training and testing.	Manually annotated each OpenI image into one of the following severity categories: borderline, mild, moderate, severe, and non-classified using the corresponding reports having cardiomegaly. (Findings, specifically cardiomegaly and their severity levels)

Table 4. Table 4: Annotator agreement.

Number of Reports	Kappa ($\kappa$)	Overall F1
	Spatial Indicator	Trajector	Landmark	Diagnosis	Hedge
First 500	0.88	0.44	0.50	0.25	0.49
Remaining 1500	0.93	0.66	0.71	0.62	0.57
Complete 2000	0.92	0.59	0.64	0.49	0.55

Table 5. Table 5: Descriptive Statistics of the annotations.

Parameter	Frequency
Average length of sentence containing spatial relation	13
Spatial Indicator	1972
Trajector	2293
Landmark	2167
Diagnosis	455
Hedge	388
Sentences containing at least 1 Spatial Indicator	1780
Maximum number of Spatial Indicator in any sentence	4
Spatial relations containing only Trajector and Landmark	1589
Spatial relations containing only Trajector, Landmark, and Diagnosis	9
Spatial relations containing only Trajector, Landmark, and Hedge	70
Spatial relations containing all 4 spatial roles	304
Spatial relations containing more than 1 Diagnosis	118
Maximum Diagnosis terms associated with any spatial relation	4

Table 6. Table 6: Average Precision, Recall, and F1 measures of 10-fold CV. Sp-In – Spatial Indicator .

Metrics	Sp-In	Trajector	Landmark	Diagnosis	Hedge	Overall
Precision (%)	$78.86$	$90.05$	$95.01$	$70.35$	$74.66$	$89.21$
Recall (%)	$99.25$	$90.55$	$94.23$	$73.15$	$72.67$	$89.25$
F1	$87.82$	$90.28$	$94.61$	$71.47$	$73.27$	$89.22$

Full text

1 Introduction

There has been a growing interest in automatically extracting useful information from unstructured reports in the medical domain. Radiology reports contain a wealth of important information and have been one of the most explored free text clinical report types for information extraction using NLP. Automatic recognition of clinically important information such as actionable findings and their corresponding location and diagnoses facilitates the time-consuming process of manual review of the reports containing radiologists’ descriptions of imaging results.

Besides radiology-specific knowledge and experience, the ability to mentally visualize complex 3D structures based on the interpretation of 2D images (e.g., X-ray diffraction images) remains vital for radiologists. A few studies [1] [2] have highlighted the possible requirement of spatial ability skills in prospective radiologists to perceive and understand the spatial relationships between different objects in radiology practice. These spatial interpretations from images are also summarized in the corresponding free text reports. Thus, radiology report texts have a high prevalence of spatial relations in the way radiologists describe imaging findings and their association with anatomical structures. These spatial relations provide sufficient contextual information to visualize a radiologist’s interpretations of the findings or imaging observations with reference to a body location. Moreover, some of these spatially-grounded findings demand immediate action by the referring physician. Therefore, it is beneficial to generate structured representations by understanding the spatial meanings from these unstructured reports, which can then be used for a variety of purposes such as easy review of the important actionable findings by the referring clinicians, analysis for predictive modeling, and automatically generating more complete annotations containing spatial and diagnosis-related information of findings for associated images.

In the general domain, earlier studies [3][4] have formulated and evaluated the spatial role labeling (SpRL) task for extracting spatial information from text by mapping language to a formal spatial representation. In the SpRL annotation scheme, an object of interest (Trajector) is associated with a grounding location (Landmark) through a preposition or spatial trigger (Spatial Indicator). For example, in the sentence, “The book is on the table”, the spatial preposition ‘on’ indicates the existence of a spatial relationship between the object ‘book’ (Trajector) and its location ‘table’ (Landmark).

In the medical domain, one study so far by Roberts et al. [5] has used the SpRL scheme to extract spatial relations between symptoms/disorders and anatomical structures from consumer-related texts. Similar spatial roles can be constructed for radiology text. For instance, in a radiology report sentence, “Mild streaky opacities are present in the left lung base”, the location of a clinical finding ‘opacities’ (Trajector) has been described with respect to the anatomy ‘left lung base’ (Landmark) using the spatial preposition ‘on’ (Spatial Indicator).

Moreover, radiologists oftentimes document potential diagnoses related to the clinical findings which are spatially grounded. Consider the following example:

• Stable peripheral right lower lobe opacities seen between the anterior 7th and 8th right ribs which may represent pleural reaction or small pulmonary nodules.

Here, presence of a finding – ‘stable peripheral right lower lobe opacities’ at a specific location – ‘anterior 7th and 8th right ribs’ may elicit the radiologist to document two possible diagnoses – ‘pleural reaction’ and ‘small pulmonary nodules’. Since the actual occurrence of a disorder is highly dependent on various patient factors such as other physical examinations, laboratory tests, and symptoms, the radiologists usually describe diagnoses with uncertainty phrases or hedges. For instance, in the example above, the hedge term ‘may represent’ is used to relate a finding and its corresponding body location with the most probable diagnoses.

Our work recognizes granular information about the interpreted diagnoses by identifying them in context to the same spatial preposition (e.g., in, of, within, around) connecting a clinical finding to an anatomical location. Thus, we extract detailed information about a finding, the body location where the finding is detected, possible diagnoses associated with the finding, and also any hedging term used by radiologists in interpreting these diagnoses. Additionally, the finding and the location terms contain their respective descriptors (e.g., the descriptor ‘mild streaky’ associated with the finding ‘opacities’).

In this paper, we propose a framework as a preliminary step to understand textual spatial semantics in chest X-ray reports. We define a basic spatial representation framework that extends SpRL for radiology (Rad-SpRL) involving interactions among common radiology entities. Since most of the actionable clinical findings in all types of radiology reports are spatially located and represent a probable diagnosis, Rad-SpRL can be easily extended to other report types. Consider the following sentence from a head CT report:

• A well circumscribed hypodense 1 cm lesion is seen in the right cerebellar hemisphere consistent with prior stroke.

Here, the spatial preposition ‘in’ describes that the finding ‘lesion’ is located inside the anatomical structure ‘right cerebellar hemisphere’ which is also consistent with the diagnosis ‘stroke’. To evaluate this representation, we manually annotated a corpus of 2000 radiology reports (a subset of publicly available OpenI chest X-ray reports [6]) using Rad-SpRL. Owing to the promising results of applying deep learning models in entity and relation extraction, we have adopted a bidirectional long short-term memory (Bi-LSTM) recurrent neural network with a conditional random field (CRF) layer as our baseline model to identify detailed spatial relationships, including diagnosis and hedging terms from the reports.

The organization of the rest of the paper is as follows. Section 2 highlights the previous relevant studies on chest radiology, spatial relation extraction, and the OpenI dataset [6]. Section 3 describes our new corpus of 2000 chest X-ray reports annotated according to Rad-SpRL and the annotation process that produced this corpus. Section 4 includes a description of our baseline system based on Bi-LSTM-CRF for extracting fine-grained spatial information, including the implementation details. Results are summarized in Section 5, while Section 6 contains a discussion of the results. Section 7 concludes the paper and provides directions for future work.

2 Related work

2.1 Types of chest radiology entities extracted

Numerous studies have focused on extracting specific information such as clinical findings or imaging observations, differential diagnoses, and anatomical locations from chest-related reports (highlighted in Table 1). In Table 1, we compare the specific information types or the radiology entities extracted in the previous studies from chest radiology reports using NLP. We primarily pay attention to the clinically-important entities which are common across various types of radiology reports. We also do not take into account the cases where uncertainty and negation information were used to detect the presence or absence of a particular finding or a disease [7]. For example, Hedge is not considered as extracted in Table 1 when the uncertainty levels are classified into negative, uncertain or positive for each finding term extracted [8]. Further, in Table 1, we have not considered studies dealing with specific body locations (e.g., mammography reports containing breast imaging information, and head CT reports) since the entities of interest are usually very domain-specific such as ‘Clock face’, ‘Depth’, ‘BI-RADS category’ etc. in case of mammography reports. We also do not take into account the works which focused on detecting a specific disease such as pneumothorax [9] or pulmonary lesion [10] from chest radiographs. Note that these works did not attempt to extract all the information types collectively, neither did they focus on identifying any association or relation among these entities.

2.2 Relation extraction including spatial relation

In context to detecting relations among radiology entities, one study by Sevenster et al. [16] has built a reasoning engine to correlate clinical findings and body locations in radiology reports utilizing the Medical Language Extraction and Encoding System (MedLEE). However, the major limitation of this work is the system’s poor recall. Yim et al. [17] has worked on extracting relations containing tumor-specific information from radiology reports of hepatocellular carcinoma patients. In Table 2, we present the two works relevant to spatial information extraction from radiology reports. The main limitations of Rink et al. [18] are the usage of appendicitis-specific lexicons and the requirement of manual effort in crafting rules based on syntactic dependency patterns to identify the spatially-grounded inflammation description. Besides being domain-specific, another limitation of Roberts et al. [19] is that the study extracts only the location entities associated with an actionable finding and this required relying on heavy feature engineering.

2.3 Studies using OpenI X-ray report annotations

OpenI is a biomedical image search engine111https://openi.nlm.nih.gov/. One of its datasets is a public chest X-ray dataset containing 3,955 de-identified radiology reports from the Indiana Network for Patient Care released by National Library of Medicine [6]. (Hereafter referred to simply as the OpenI dataset.) We have presented an example of the manual annotation of a sample report in the OpenI dataset in Figure 1 (the annotations are inspired by MeSH terms). Although most of the OpenI annotations embody the relationship between finding and location, there are, however, a few missing relations. For example, note that in Figure 1 the OpenI manual annotations contain the normalized finding Pulmonary Emphysema corresponding to the phrase ‘emphysematous changes’ in the report, but do not annotate the associated location ‘right upper lobe’. The OpenI dataset has been used previously in many studies, presented in Table LABEL:table:openi_dataset_studies. However, most of the these studies focused on the extraction of only the disease/finding [7], [20], [21], [22], [23]. Two studies worked on automatically annotating both disease and disease descriptions (e.g., location, severity) [24], [25] similar to the human annotations in Demner-Fushman et al. [6]. However, all these works ignored distinguishing diagnosis terms from findings (except for Peng et al. [21]), and annotating correlations between them. We describe annotation-specific limitations of each of these works in Table LABEL:table:openi_dataset_studies.

3 Proposed spatial relation annotation framework

3.1 Dataset for spatial relation annotation

A subset of 2000 reports from a total of 2470 non-normal reports as judged by two human annotators in Demner-Fushman et al. [6] was used to create our spatial relation corpus. This newly annotated chest X-ray corpus contains spatial relations between findings and body locations as well as the correlated probable diagnoses and the hedging terms used in qualifying the diagnoses. We have presented a simple comparison between the OpenI manual annotations and our spatial annotations of a sample report in Figure 1. Note that we have not annotated other findings appearing in the report such as Opacity and Pulmonary Fibrosis as their corresponding body locations are not described through any spatial preposition.

3.2 Rad-SpRL

Our spatial representation framework (Rad-SpRL) consists of 4 spatial roles (Trajector, Landmark, Hedge, and Diagnosis) with respect to a Spatial Indicator. The spatial roles and the Spatial Indicator are defined as follows:

Spatial Indicator: term (usually a preposition, e.g., in, within, at, near) that triggers a spatial relation 2. 2.

Trajector: object (finding, anatomical location) whose spatial position is being described 3. 3.

Landmark: location of the Trajector (may also be chained as a Trajector to another Landmark) 4. 4.

Hedge: phrase indicating uncertainty (e.g., could be, may represent), generally in reference to the Diagnosis and very rarely in the Trajector 5. 5.

Diagnosis: disease/clinical condition the radiologist associated with the finding

In most of the cases where a sentence contains spatial information, a finding (Trajector) is usually detected at a particular body location (Landmark) where the Trajector term appears to the left of the Spatial Indicator and the Landmark to its right. However, there are instances where a spatial preposition describes the body location (Landmark) with its associated abnormality (Trajector) and the Trajector term appears to the right of the Spatial Indicator and Landmark to the left (refer to example in Figure 2(d)). We have presented a few specific examples to highlight how various spatial roles and Spatial Indicators are identified in sentences following the above definitions of Rad-SpRL in Figure 2. Please note that we have considered disease/condition terms as Diagnosis only when they are documented in conjunction with any spatially-located finding, or in other words are entirely probable diagnoses inferred from the finding. Also note that there is some ambiguity between a finding and a diagnosis, such that the same phrase may appear as a Diagnosis in one relation while being a Trajector in another. Our purpose here is not to formally distinguish between a finding and a diagnosis, but rather to identify the spatial relationships in radiology reports where the Trajector is generally a finding (or artifact in the image) and the Diagnosis is generally a well-understood disease term.

3.3 Annotation process

Two annotators (S.E.S., a medical librarian, and L.R., an MD) annotated the spatial roles for each identified Spatial Indicator in each of the 2000 reports independently. They also were the annotators that manually coded the findings/diagnoses available as part of OpenI dataset [6]. The spatial relation annotations were conducted in two rounds and reconciled after each. The first round consisted of annotating the first 500 reports and the second round consisted of annotating the remaining 1500. Figure  3 shows a sample annotated report from the corpus.

3.3.1 Annotation agreement

The inter-annotator agreement statistics for both Spatial Indicator and spatial roles are shown in Table 4. The Kappa ($\kappa$) agreement between the two annotators has been calculated for Spatial Indicator (as this is a binary classification task) whereas we report the overall F1 agreement for annotating the spatial role labels (as this is a role identification task). The Kappa agreement is high for Spatial Indicators in both annotation rounds. The F1 agreements for the 4 spatial roles are fairly low in the first round with much improvement in the second round. This is mainly because it is relatively easy and unambiguous to locate a spatial preposition in a sentence compared to identifying the spatial roles. All conflicts were reconciled with an NLP expert (K.R.) following each round of annotation. The moderate agreement rate for Trajector and Diagnosis roles was likely due to ambiguity in distinguishing the two roles in a sentence, especially when the language pattern is different from the usual. Consider the examples below:

Probably scarring in the left apex, although difficult to exclude a cavitary lesion. 2. 2.

There are irregular opacities in the left lung apex, that could represent a cavitary lesion in the left lung apex.

In the first example, ‘scarring’ was annotated as a Trajector after reconciliation as its spatial location (‘left apex’) is described directly, although there is a higher chance of annotating it as a Diagnosis since most of the probable diagnoses terms are usually preceded by a Hedge term (‘Probably’ in this case). Similarly, ‘cavitary lesion’ is indirectly connected to the same body location (‘left apex’) and has been interpreted as an additional finding. So, ‘cavitary lesion’ was also annotated as a Trajector and not as a Diagnosis. In the second example, ‘cavitary lesion’ was annotated as a Diagnosis in context to the first ‘in’ in the sentence, whereas the same term ‘cavitary lesion’ was annotated as a Trajector when its role was identified in context to the second ‘in’. As previously noted, this difference where the same term can be both a Trajector and Diagnosis in different sentences is a consequence of focusing on explicitly representing the spatial language as described as well as the natural ambiguity between a finding and diagnosis in radiology. As a result, some downstream processing or interpretation is still required, which we leave to future work.

3.3.2 Annotation statistics

A total of 1972 spatial relations are annotated in our corpus of 2000 reports. Most of the Trajector terms were findings, however, 176 out of 2293 terms annotated as Trajectors were anatomical locations (example shown in Figure 2(c-2)). 118 Spatial Indicators had more than one probable Diagnosis, out of which 98 were associated with 2 Diagnosis terms, 17 were associated with 3 Diagnosis terms, and 3 had 4 associated Diagnosis terms. There are 1062 reports containing at least one sentence triggering a spatial relation. In 1062 reports, there are 1780 sentences each containing at least one Spatial Indicator (1559 sentences containing exactly one Spatial Indicator and remaining 221 containing more than one Spatial Indicator). We have highlighted a brief descriptive statistics of our corpus based on the reconciled version of the annotations in Table 5.

4 Methods

4.1 Model for spatial relation extraction

We formulate the spatial role extraction as a sequence labeling task. We utilize a Bi-LSTM CRF framework similar to the proposed architecture in Lample et al. [29] for spatial role labeling. The main intention behind using Bi-LSTM is that it works well for taking in the long term dependencies in a sentence, and the bi-directional sequential architecture adds more benefits by considering both the previous and future context of a word. The CRF in the decoding layer takes into account the sequential information in the sentence while predicting the sequence labels related to any spatial role (Trajector/Landmark/ Diagnosis/Hedge). We utilize a Bi-LSTM network to incorporate character embedding $x_{i}^{ce}$ (where each character is denoted $c_{i,j}$) for each word $w_{i}$ in a sentence where $i$ represents the word position. This is to better handle out-of-vocabulary, rare, and misspelled words. For every word, this character embedding is then concatenated with the respective pre-trained word embedding $x_{i}^{we}$. In addition, a Spatial Indicator embedding $x_{i}^{ind}$ is concatenated to the word and character embeddings to distinguish the indicators from non-indicator words. The final concatenated representation [$x_{i}^{we}$; $x_{i}^{ce}$; $x_{i}^{ind}$] is fed into the final Bi-LSTM network with one hidden layer. The overall architecture is presented in Figure 4.

4.2 Pre-processing

We preprocess the Rad-SpRL dataset to generate input sequence for the model. For each Spatial Indicator in a sentence, we create an instance or sample of the sentence. This results in a total of 1867 sentences with unique position of the Spatial Indicator. For each instance, we tag all the spatial roles (Trajector/Landmark/Diagnosis/Hedge) as well as the Spatial Indicator. Creating separate sentence instance for each Spatial Indicator helps in dealing with cases where the same word can be both a Trajector and a Landmark in context to two different Spatial Indicators in the sentence (example shown in (c-1) and (c-2) in Figure 2). Also, annotating only the roles associated with a single Spatial Indicator provides the model unambiguous information about the position of the specific indicator term to which these roles are associated. We follow Beginning (B), Inside (I), and Outside (O) tagging scheme and label the Spatial Indicator term as Indicator. The input to the final Bi-LSTM neural network consists of words and the corresponding B, I, O labels for a set of sentences. The following example shows the tagged words for the sentence – “Minimal degenerative changes of the thoracic spine”.

• $[$Minimal]${}_{\textsc{B-Trajector}}$ [degenerative]${}_{\textsc{I-Trajector}}$ [changes]${}_{\textsc{I-Trajector}}$
• [of]${}_{\textsc{Indicator}}$ [the]${}_{\textsc{O}}$ [thoracic]${}_{\textsc{B-Landmark}}$ [spine]${}_{\textsc{I-Landmark}}$

4.3 Experimental settings and evaluation

10-fold cross validation was performed to evaluate the performance of the Bi-LSTM CRF model in extracting the spatial roles given the gold Spatial Indicator in a sentence. The training, validation, and the test sets consisted of 1495, 186, and 186 sentences (80%/10%/10% splits). We reported the average Precision, Recall, and F1 measures of each of the 4 spatial roles across 10 iterations. We also calculated the overall measures of the three metrics considering all the roles collectively. Exact match was performed for evaluating the model performance on the test set. In addition to the spatial roles, we also evaluated the model performance in extracting the Spatial Indicators using the same folds.

We used pre-trained medical domain MIMIC-III word embeddings of 100 dimensions 222https://northwestern.app.box.com/s/eprxyxmee37p3d6khqbpn125tyttq4u6 in our experiments. The character and the indicator embeddings were initialized randomly and altered during training. The dimensions of character and indicator embeddings were 100 and 5 respectively. The model was implemented using TensorFlow [30], and the hyperparameters were chosen based on the validation set. LSTM hidden size was set at 500, dropout rate was set at 0.5, learning rate at 0.01, and learning rate decay at 0.99. We used Adam optimizer and trained the model for a maximum of 20 epochs.

5 Results

The results of the 10-fold CV for both spatial role extraction and the Spatial Indicator extraction from the Rad-SpRL corpus are presented in Table  6. For spatial role extraction, we considered the gold Spatial Indicator in a sentence. As the results demonstrate, the model performed the best in case of Landmark with an average F1 of $94.61$, followed by Trajector where the average F1 is $90.28$. Using the same Bi-LSTM-CRF architecture, we found that the recall in case of Spatial Indicator extraction is high ($99.25$%), which suggests that most of the gold indicators were predicted correctly by the system. However, the relative low precision ($78.86$%) demonstrates that many prepositions in the dataset were wrongly classified as Spatial Indicator, which would consequently generate false positive spatial roles if these predicted indicators are used instead of the gold ones for spatial role labeling.

6 Discussion

In this paper, we extract the four spatial roles with respect to a Spatial Indicator in a sentence following the Rad-SpRL annotation scheme. This includes identifying the probable diagnoses with associated hedges in context to a spatial relation between any finding and its associated location. The results in Table  6 demonstrate that the Bi-LSTM-CRF model achieves promising results in extracting the spatial roles from the Rad-SpRL corpus. The average F1 measures are around 90 and 95 for extracting Trajector and Landmark respectively. However, the F1 measures are comparatively low for Diagnosis and Hedge roles. The reason behind this can be attributed to the lesser number of Diagnosis and Hedge terms in the dataset (5 to 6 times lesser than both Trajector and Landmark terms) and greater distance between the Spatial Indicator and the Diagnosis/Hedge terms compared to the Trajector/Landmark terms.

Taking into account the relatively low F1 measure for Diagnosis, we analyzed the errors and many of these are due to the challenging nature of the description used by radiologists to express their interpretation. For example, in the sentence – “Low lung volumes with bibasilar opacities may represent bronchovascular crowding.”, the Diagnosis ‘bronchovascular crowding’ is falsely classified as a Trajector. This might be because there are many instances in the dataset where ‘bronchovascular crowding’ appears as Trajector, as often a Diagnosis term itself appears in a spatial relationship.

In this work, we have considered both positive and negative spatial relations. We aim to differentiate the negated relations in future. Future work should also be directed toward building an end-to-end system based on neural joint learning models [31, 32] that would extract both Spatial Indicator and the spatial roles together. We further aim to consider non-prepositional spatial expressions as Spatial Indicators (e.g., verbs such as ‘demonstrates’, ‘shows’ etc.) that indicate the presence of any spatial relation between finding and body location. Besides clinical findings, we also intend to extend the Rad-SpRL framework to extract other important and common spatially-grounded radiology entities such as medical devices from the reports. Following example illustrates a sample sentence where the medical device ‘Right IJ venous catheter’ acts as the Trajector in context to its associated location ‘proximal SVC’ that acts as the Landmark:

• Right IJ venous catheter terminates at the proximal SVC.

Another limitation of this work is that the inter-sentence spatial relations are not covered, although the frequency of such cases are very rare in the Rad-SpRL corpus. We also aim to evaluate the generalizability of our sequence labeling method in extracting the spatial roles from datasets across institutions. For standardization of the extracted spatial roles, we further aim to normalize them utilizing the existing radiology lexicons such as RadLex [33] codes. From a method perspective, we plan to apply more recent deep learning methods such as BERT-based models [34], highway networks [35], and syntax-enhanced models [36] to improve the existing performance of spatial role extraction from the Rad-SpRL corpus.

7 Conclusion

This paper proposes a spatial representation framework in radiology (Rad-SpRL). It provides a detailed description of the annotation scheme used for extracting spatial information from radiology reports. This consists of annotating four radiology-specific spatial roles in a dataset of 2000 chest X-ray reports. The spatial roles are annotated in the context of a Spatial Indicator which denotes the presence of a spatial relation between clinical findings and body locations. It additionally identifies probable diagnoses and hedging terms associated with the spatially-related finding-location. For this, we employ a Bi-LSTM-CRF model to automatically extract the spatial roles from our annotated Rad-SpRL corpus. The model achieves satisfactory performance with average F1 measures of around 90, 95, 71, and 73 for Trajector, Landmark, Diagnosis, and Hedge roles, respectively. In the future, we aim to evaluate the model on much larger datasets, improve the current performance by using more recent and advanced deep learning techniques, and develop joint learning models for extracting the Spatial Indicator and the spatial roles jointly.