Attentional Road Safety Networks

TL;DR

This paper introduces a region guided attention network with domain adaptation for road safety classification from satellite images, achieving high accuracy and outperforming subjective analysis.

Contribution

It proposes a novel region guided attention network combined with a domain adaptation approach and a covariance-based loss function for improved road safety mapping.

Findings

Achieves 86.21% accuracy on London dataset.

Increases accuracy by 4% on NYC with domain adaptation.

Outperforms subjective analysis by 23.12%.

Abstract

Road safety mapping using satellite images is a cost-effective but a challenging problem for smart city planning. The scarcity of labeled data, misalignment and ambiguity makes it hard for supervised deep networks to learn efficient embeddings in order to classify between safe and dangerous road segments. In this paper, we address the challenges using a region guided attention network. In our model, we extract global features from a base network and augment it with local features obtained using the region guided attention network. In addition, we perform domain adaptation for unlabeled target data. In order to bridge the gap between safe samples and dangerous samples from source and target respectively, we propose a loss function based on within and between class covariance matrices. We conduct experiments on a public dataset of London to show that the algorithm achieves significant results with the classification accuracy of 86.21%. We obtain an increase of 4% accuracy for NYC using domain adaptation network. Besides, we perform a user study and demonstrate that our proposed algorithm achieves 23.12% better accuracy compared to subjective analysis.

Tables

Table 1. Table 1 . Sample traffic accident reports for London.

ID	Date	Time	Day of the week	Latitude	Longitude	No. of Vehicles	No. of Casualties
1	01/11/2016	2:30:00 AM	3	51.5847	0.2793	2	1
2	21/11/2016	6:00:00 PM	2	51.5092	0.0472	2	2
3	20/05/2016	7:00:00 PM	6	53.8126	-2.9323	1	1
4	11/01/2016	7:07:00 AM	2	54.9785	-1.6203	2	3

Table 2. Table 2 . The comparison of the London and NYC in terms of area and population density.

	London	NYC
Area (mi2)	607	468.5
Population Density (people per mi2)	14,550	28,491

Table 3. Table 3 . The complete description of the user-study outcome (in %).

		Predicted
		Safe	Dangerous	Unsure
Actual	Safe	40.41	05.56	4.32
	Dangerous	26.80	22.37	0.51

Table 4. Table 4 . Comparison of classification accuracy (in %) for ResNet50, VGG19, and variants of the DAM both for original dataset (London) and cross dataset (NYC and Denver).

Model	Original Data	Cross Data
	London	NYC	Denver
ResNet50	85.77	69.16	70.00
VGG19	85.83	64.60	70.00
DAM (HS)	85.81	72.28	76.20
DAM (VS)	85.52	74.77	75.00
DAM (SQ)	85.86	70.70	70.00
DAM (HS+VS)	85.34	70.37	70.01
DAM (HS+VS+SQ)	86.21	67.23	69.86

Table 5. Table 5 . A comparison of false positive rate between DAM and ResNet50. All results are in percentage.

Model	London	NYC	Denver
DAM	07.60	22.05	29.20
ResNet50	12.73	39.05	40.80

Table 6. Table 6 . A comparison of DAM, DAM-DA with L D ​ A subscript 𝐿 𝐷 𝐴 L_{DA} , and DAM-DA with L D ​ A S ​ T superscript subscript 𝐿 𝐷 𝐴 𝑆 𝑇 L_{DA}^{ST} network with batch-size of 16.

Model	Accuracy (%)	FPR(%)
DAM	71.94	30.03
DAM$-$DA$-L_{DA}$	75.75	18.94
DAM$-$DA$-L_{DA}^{ST}$	74.73	30.69

Table 7. Table 7 . A comparison of the performance of our proposed deep learning model (DAM) and humans.

Model	Acc(%)	Precision	F1
Humans	62.78	0.601	0.687
DAM	85.90	0.909	0.850

Equations

$$\begin{array}{r}{\sum }_{SW}={\sum }_{i,j\ne i}(x_i-x_j)(x_i-x_j{)}^T\\ +{\sum }_{i,j\ne i}(y_i-y_j)(y_i-y_j{)}^T\end{array}$$

$${\sum }_{SB}={\sum }_{i,j}(x_i-y_j)(x_i-y_j{)}^T$$

$$L_{DA}={\Vert {\sum }_{SW}-{\sum }_{TW}\Vert }_F^2+{\Vert {\sum }_{SB}-{\sum }_{TB}\Vert }_F^2$$

$$L_{DA}^{ST}=\frac{1}{d}{\Vert C_S-C_T\Vert }_F^2$$

Full text

Road safety, Region guided network, Domain Adaptation

††copyright: none

1. Introduction

Road accidents remain one of the pressing communal welfare concerns. Regardless of notable advancements in the field of vehicle technology and road engineering, on a global scale, traffic accidents are one of the leading causes of premature death and injury. According to official statistics from the World Health Organization (WHO), more than 1.25 million people die every year due to traffic accidents. Besides, traffic accidents cost many countries up to 3% of their GDP (Organization, 2015). Therefore, minimizing the road accidents is a worldwide challenge and can benefit a majority of the nations in different ways. Towards this, it is important to understand which road segments are potentially dangerous or safe.

Few works have shown the influence of environmental factors like weather, light condition on road accidents (Tamerius et al., 2016; Chung et al., 2018; Black and Villarini, 2018). However, gathering such data is costly and laborious. Further, due to lack of resources and technology, such data is not maintained properly in most low and middle-income nations and unfortunately, these are the nations which suffer dreadfully from traffic accidents (Schmucker et al., 2010). Hence, there is a need for an efficient approach which can work well with easily available and affordable data. To this end, satellite images are used for road safety mapping (Najjar et al., 2017). However, the dataset is mostly imbalanced, and percentage of the safe class is far more compared to the dangerous class. Further, the images are misaligned. Thus, using such data for training efficient supervised models is another challenge.

In addition, it has been observed that models trained using classical machine learning techniques for one region do not perform well if tested on regions that differ immensely in terms of traffic regulations, city planning, architecture, etc. (Najjar et al., 2017). Also, it is quite inconvenient to obtain traffic accident data for every different region and train a model for the same. Therefore, the difference in domains is yet another challenge in the effective road safety mapping.

To address these challenges, we propose a deep region guided attention network as shown in Figure 1. We use a base network (ResNet-50) to extract global features. We further use a sub-network which can attend to individual subregions. Towards this, we extract the conv 2 layer features and divide them into $N$ non-overlapping regions. Region or part based networks have shown good accuracy in re-identification tasks (Zhu et al., 2017). We further augment this network to adapt to different domains. We use a loss function based on the covariance matrix to minimize the gap between the source and target domains. Our contributions are:

• •

We propose a deep learning framework that uses the region guided attention network to predict accurate city-scale safety maps from satellite images.

• •

We propose a domain adaptation network with a training loss which minimizes the Frobenius norm of the difference of within class covariance matrices of source and target, as well as the difference of between class covariance matrices of source and target.

The rest of this paper is organized as follows. The related work is discussed in section 2. Section 3 presents the data sources and collection technique and also introduces the formulation of the problem. Our approach is explained in section 4. Section 5 shows a user-study followed by Section 6, which describes our experiments. Finally, the paper is concluded in section 7.

2. Related Work

In this section, we briefly outline an overview of studies on accident hotspot detection (also known as, black spots and black zones), understanding road accidents, and lastly, the city-scale road-safety mapping that exploits recent advancements in the field of deep learning.

There are few studies on the detection of high-density accident hotspots. Kernel density estimation and clustering have been used to create classifiers to identify accident hotspots (Anderson, 2009; Bíl et al., 2013). Another interesting work includes the use of Bayesian statistics to predict accidents (Deublein et al., 2013). Fawcett et al. presented a Bayesian hierarchical model to rank accident hotspots in line with their possibility to surpass a threshold accident count in some future time period. Researchers have developed models to estimate the total number of crashes, the number of injury crashes, and the number of property damages and proved that the models are statistically meaningful and closer to real-world data (Pulugurtha et al., 2013). In addition, association rules have been used for the identification of accident situations that frequently occur together (Geurts et al., 2003, 2005; Das et al., 2018). Gomes proposed a model for accident frequency estimation which takes the influence of the road characteristics into account.

In the recent past, the remarkable progress in deep learning has contributed significantly to the field of computer vision. Spatio-temporal data have been used by researchers to predict the number of accidents in a given area with the help of ConvLSTM (Yuan et al., 2018). Chen et al. developed a Stacked Denoising Autoencoder for prediction of traffic accident risk level at the city-scale using real-time GPS data of users. In a similar study, Najjar et al. demonstrate that visual attributes captured in satellite image can be used as a proxy signal of road safety. They proposed a deep-learning based mapping framework that exploits open data to predict city-scale safety maps at bearable costs. However, the distinction among the above frameworks resides in their application itself. Both (Chen et al., 2016; Yuan et al., 2018) are interested in the real-time prediction of traffic accidents, whereas (Najjar et al., 2017) is inclined towards assisting in informed decision-making for city-planning and policy formulation when heterogeneous data is not accessible or bearable. Our work is closer in spirit to Najjar et al.; however, there are three major differences. First, we only consider safe and dangerous classes compared to an additional neutral class defined by Najjar et al. This is because a neutral road segment may not assert a strong decision making during road safety planning. Second, the authors extract features from AlexNet, whereas, we propose a novel architecture. Third, we also perform domain adaptation for the unlabeled target domain. Our two major goals are as follows. First, we propose a region guided attention network to generate a city-scale safety map. Second, we propose a domain adaptation technique to generate safety maps where traffic accident data is not available.

Domain Adaptation is a branch of machine learning in which we aim at learning a well performing network from a source data that can generalize well on a different but related target data distribution (Ben-David et al., 2010). It has been widely studied in visual applications (Csurka, 2017; Wu et al., 2018). One of the popular class of algorithms is discrepancy based methods (Tzeng et al., 2014; Long et al., 2015; Sun and Saenko, 2016). In particular, Sun and Saenko propose to minimize the Frobenius norm of the difference between the feature covariance matrix of source and target. In our case, we also make use of the labels available for source and pseudo-labels for the target. These pseudo-labels are determined by the model trained on the source data. We then compute the feature covariance matrix for safe and dangerous classes respectively. Further, we compute our domain adaptation loss function based on within and between class covariance matrices of source and target.

3. Overview

In this section, we present the data that we use at various stages of the work. We also introduce the formulation of our problem.

3.1. Data

We select two cities, London (UK) and New York City (USA) for the purpose of this study. Our reason for selecting these two cities is two-fold. First, the availability of sufficient data to work on and second, the extreme difference in traffic regulations, city planning, architecture etc. Figure 2 shows the difference in the street network orientation of both the cities (Boeing, 2018). All the data that we use in this work are available as open data. Open data is the scheme under which some datasets are freely available to use and republish, without any restrictions (Dietrich et al., 2009). We also present results for Denver (USA) which is altogether a different domain.

Traffic Accident Data: We collected 99,516 traffic accident records for London from 2013 to 2016.111https://www.europeandataportal.eu/data/en/dataset/road-accidents-safety-data We gathered 1,256,205 traffic accident reports for New York City (NYC) from 2012 to 2017.222https://opendata.cityofnewyork.us Similarly, we collected 143,776 traffic accident reports for Denver from 2013 to 2017.333https://www.denvergov.org/opendata/dataset/city-and-county-of-denver-traffic-accidents Each record has numerous attributes such as latitude, longitude, date, time, vehicle type etc. However, there is a difference in the datasets. Table 1 presents sample records from London data. Information such as junction type, weather etc. is only available in London dataset. Therefore, we restrict ourselves to common attributes only.

Satellite Images: We collect satellite images using Google’s Static Maps API.444https://cloud.google.com/maps-platform/

3.2. Problem Formulation

We divide the whole study area by imposing a grid G, where each grid $g_{i}$ is a square region of $s\times s$. Here, $s$ is equal to $30$ meters which is a sufficient area for traffic accident analysis (Najjar et al., 2017). In this work, our objective is to produce a model that learns features such that it can predict the safety level of a given grid $g_{i}$ using its raw satellite image.

In order to curate a training dataset, we plot each accident on the grid using the locations from the Traffic Accident data. Each grid $g_{i}$ is given a safety score $S_{i}$ which is equal to the number of accidents in the grid. The higher $S_{i}$ corresponds to higher number of accidents.

We obtain satellite image for each of these grids. To obtain the image label for each $g_{i}$, we apply the classical k-means algorithm to bin $S_{i}$ into two clusters. Hence, we get two clusters of safe and dangerous locations. We observe that the obtained set of labels are highly imbalanced. Over 88% of the regions are labeled as safe. Therefore, we re-sample the dataset by down-sampling the majority class such that both classes are balanced out.

4. Deep Attention-Network Approach

In this section, we begin by introducing the network architecture. Then we elaborate the technique used to implement the unsupervised domain adaptation for satellite images that belong to different domains.

4.1. Network Architecture

The model consists of a global and a local network, with the global network borrowing layers from ResNet50 (He et al., 2016). We show the architecture of the proposed model in Figure 3. There are three 2-way switches in the architecture which enables the network to run in two different modes. When all the switches are connected to red nodes, the Deep Attention Model (DAM) mode is enabled whereas if switches are connected to all the green nodes, the Deep Attention Model with Domain Adaptation (DAM-DA) mode is enabled. The solid and dashed lines represent DAM and DAM-DA mode respectively. We discuss the DAM-DA mode in the next section. As shown in Figure 3, the global network is augmented with a local network. In our experiments, we notice that the global network based on ResNet only, gives significant false positives, where false positives are defined as dangerous regions being predicted as safe. A possible reason behind such false positives could be that the satellite images are usually misaligned. To overcome misalignment and improve the performance of ResNet, we design the network such that it can exploit local regions in addition to the global region. We perform Region of Interest (ROI) pooling on conv 2 layer of the model with the subregions being $N$ non-overlapping blocks. We pass features of each subregion through a local network. The local network consists of conv${}_{1}^{l}$ and conv${}_{2}^{l}$ layers. Each subregion of size 7x7 acts as an input to the local network to extract features and to identify which of them is useful for the classification task. Thus, this network attends to the regions which can significantly contribute towards enhancing the classification accuracy. We perform average pooling on the output of conv${}_{2}^{l}$ layer. We obtain a feature vector of size 4096x1x1 that is passed through $fc_{l}$. The subregion with the maximum prediction probability of either of the two classes (Safe and Dangerous) is concatenated to the feature-map of the global network. We perform concatenation after the conv 4 layer of the global network. Thus, we can see that only certain subregions can guide the global feature vector for classification. Once average pooling is performed, we pass the feature vector through $fc$ to obtain the final $d$-dimensional vector. We train the network using cross entropy loss $(L_{C})$.

4.2. Domain Adaptation

In this section, we discuss our proposed Domain Adaptation (DA) technique for the target domain where there is no annotated data available. In our experiments, we consider London as the source domain for which labeled data is present, whereas, NYC as the target domain for which labels are not given. There are significant differences in both the cities. Although NYC is nearly 20% smaller in terms of area, it’s population density is almost double than that of London. According to Nieminen et al., higher the population density, higher are the chances of occurrence of accidents. Due to above-mentioned reasons, these cities would be a good extreme example for performing DA and can give us an idea of lower bound of model’s performance.

First, we train DAM network using data from the source domain as explained in the section 4.1. Using this trained model, we generate pseudo labels for data from the target domain. Thus, we have labels for the source domain (London) and pseudo labels for the target domain (NYC). Now, we use data from both source and target domain to train the augmented DA (DAM-DA) network as shown in Figure 3. There are three differences when compared to DAM. First, this network has two additional convolutional layers in the global network, conv${}_{1}^{g}$ and conv${}_{2}^{g}$ layers of dimensions 512x7x7 and 256x7x7 respectively, to reduce the dimensions of the feature maps. This augmentation is necessary to be able to compute the covariance matrix efficiently. We explain the utility of the covariance matrix in the later part of this section. Second, instead of conv${}_{1}^{l}$ and conv${}_{2}^{l}$ layers in the local network, we use conv${}_{1}^{l^{\prime}}$ and conv${}_{2}^{l^{\prime}}$ layer of size 512x7x7 and 256x7x7 respectively. Again, this is done to reduce the dimensions for covariance matrix calculation. Third, in DAM mode, we train the network using $L_{C}$ only, but in DAM-DA mode we use domain adaptation loss ($L_{DA}$) along with $L_{C}$. We explain $L_{DA}$ next.

Let $x_{i}$ denote the feature for i-th sample classified as dangerous in source domain. Similarly, let $y_{i}$ be the feature for i-th sample classified as safe in source domain. These features are obtained from the final layer $(fc)$ of the network as shown in Figure 3. Now, we can obtain the within class covariance matrix $\sum_{SW}$ $\in\mathcal{R}^{d\times d}$ as,

[TABLE]

Similarly, we can compute the between class covariance matrix $\sum_{SB}$ $\in\mathcal{R}^{d\times d}$ as,

[TABLE]

We can compute the within and between class covariance matrices for target domain in a similar manner. Let these be denoted by $\sum_{TW}$ and $\sum_{TB}$ respectively. Then, we use the following loss function to adapt to the target domain,

[TABLE]

where $\left\lVert.\right\rVert_{F}$ denotes the Frobenius norm. Though the pseudo labels are noisy, it is still beneficial to use them. In our experiments, we demonstrate that using the loss to minimize the distance between feature covariance matrices between source and target, which is agnostic of source labels, achieves sub-par performance when compared to $L_{DA}$.

5. User Study

We conduct a study with an aim to observe that how accurately humans can classify raw satellite images as safe or dangerous. It’s first of its kind user study. We develop an annotation portal and provide the raw satellite images from our test-data to the users. The portal has two sections. In the first section, we provide four sample images from each category. In the second section, an image is presented to the user along with three options to choose from. A user can mark an image as dangerous, safe or unsure. We show the same image to three different users and record their response to maintain the confidence in the user’s response. We also add sixteen more sample images from each class to help the user. We record users’ response for 1,000 images having 500 images from each category, i.e., safe and dangerous classes.

In this study, 38 users participated. All the participants are above 18 years of age. To check inter-annotator agreement, we compute Fleiss’ kappa (Fleiss and Cohen, 1973). We achieve 0.2743 which depicts a fair agreement. To assign a final label to the image, we calculate the mode of all three responses. If all the responses are different, we discard that response. In our study, we found 0.03% of such responses and discarded them. Finally, 67.22%, 27.94%, and 4.84% images are marked as safe, dangerous, and unsure respectively by users. In Table 3, we provide a complete description of the user-study results. With 80.32% accuracy, users are able to detect safe images whereas the accuracy drops to 45.02% for dangerous images. Therefore, in this case, we find that humans can identify safe locations with better accuracy than dangerous locations.

6. Experiments

We initially performed the experiments with AlexNet, VggNet, DenseNet, and ResNet and found that ResNet gives the best accuracy. We now prove the efficacy of the proposed technique by assessing the classification accuracy of the predicted road safety maps in multiple scenarios. It is also important to note the non-triviality of learning from satellite data, as it is seen in the user study, and can also be seen in the examples shown in Figure 4 and Figure 5.

Training Parameters: We train our network on London dataset. There are 4,517 training samples and 903 validation samples in each class. The models are trained with a batch size of 4, with learning rate as 0.0001 and a learning rate decay of 0.5 per 10 epochs. We train it over 50 epochs on a Nvidia GeForce TITAN X GPU.

Evaluation of DAM: To assess the model, we test on unseen data from London. As shown in Table 4, we test the models in multiple scenarios such as with horizontal (DAM + HS), vertical (DAM + VS), and square boxed (DAM + SQ) subregions and with the combinations of them. We test another variation of the model where the activation maps are produced based on an intermediate convolutional layer. We experiment with two conv layers; conv 2 and conv 3 layer. We finally use conv 2 layer as we achieve better results with it. We find that the DAM using HS+VS+SQ subregions outperformed ResNet50, VGG19, and its other variants on the same dataset as shown in Table 4.

One of our goals is to minimize the false positives. This is necessary as the cost of predicting safe as dangerous may only lead to overly conservative approaches for planning, whereas, having high false positive rate with more dangerous locations being marked safe can lead to fatal accidents. Therefore, we consider the classification of dangerous images as safe to be more costly than vice-versa. We test ResNet50 and DAM on London, NYC, and Denver test-set of 7,228, 8,342, and 500 images respectively. From Table 5, we can see that DAM gives fewer false positives in comparison to ResNet50 for every domain.

Cross Dataset Testing: We also perform cross-data testing and test our model on NYC and Denver dataset. From Table 4, we can see that DAM + VS and DAM + HS performs the best for NYC and Denver dataset respectively. For DAM (HS+VS+SQ), the base model trained on London with three sub-regions is over-fitting, and the network does not generalize well on Denver. When we decrease sub-regions to 2, the accuracy increases for NYC and Denver, and the model performs the best when used with only one sub-region.

Qualitative Analysis: In order to understand the behavior of the network qualitatively, we generate Class Activation Maps (CAM) (Zhou et al., 2016). In Figure 5, the images in the top row correspond to a dangerous image and its CAMs for ResNet50, DAM-Global Network, and DAM-Local Network. Similarly, the images in the bottom row correspond to a safe image and its corresponding CAMs. From the top row in Figure 5(c) and Figure 5(d), we can establish that the DAM not only identifies the green region around the divider on the road but also identifies the cars and roads in the surrounding areas as well, which positively contributes towards the identification of this image as dangerous. This image contains a dangerous location, but, ResNet50 mis-classifies it as safe. As from top row in Figure 5(b), ResNet50 seems to have identified the green region and used that to classify the sample as safe, whereas our proposed DAM is efficient enough to correctly identify it as a dangerous location. Similarly, from the CAM in the bottom row, we observe that ResNet50 recognizes only roads whereas the DAM identifies houses and trees in the image. Therefore, DAM correctly identifies this sample as safe whereas ResNet50 mis-classifies it as dangerous.

Evaluation of DAM-DA Network: We train our DAM-DA network with 2,085 images of each class, i.e., Safe and Dangerous. We use 4,170 images each from the source (London) and target (NYC) domain. We use a batch size of 16, a learning rate of 0.0001 and a decay of 0.5 per 10 epochs. It is trained for over 50 epochs. We test the DAM-DA network on 3,336 images from NYC. As shown in Table 6, DAM-DA network gives an accuracy of 75.75% in comparison to DAM network which gives an accuracy of 71.94%.

In another setting, we replace $L_{DA}$ with the loss (Sun and Saenko, 2016) in Equation 4,

[TABLE]

where $C_{S}$ and $C_{T}$ denotes the feature covariance matrix of source and target samples in a training batch, respectively. When compared to $L_{DA}$, $L_{DA}^{ST}$ does not take the class labels into account. As shown in Table 6, we obtain an accuracy of 74.73%. Similarly, DAM$-$DA$-L_{DA}$ results in the least false positives rate (18.94%) in comparison to its counterparts. Thus, we can see that the class wise covariance loss works better in this scenario.

Comparison of DAM and Humans: We test DAM on the set of same 1000 images that we use for user study. We present these results in Table 7. We can see that DAM is more accurate than humans by a significant difference of 23.12%. Moreover, there is a substantial difference of 30.77% in precision. Therefore, this proves that DAM performs better than humans in this case.

7. Conclusion

In this paper, we address the challenge of learning efficient embeddings to classify the road segments as dangerous or safe using easily available and inexpensive data. We leverage open data and satellite images to predict city-scale road safety maps. We propose a deep learning based model that uses a region guided attention network. It consists of a global and a local network. The local network attends to features in the subregions and the features with maximum prediction score are used to guide the global features to enhance the accuracy. We evaluate our network on the public dataset of London and achieve the accuracy of 86.21%. We experiment with the cross-datasets of NYC and Denver and achieve significant results with the accuracy 74.77% and 76.20% respectively. In addition, we propose a covariance loss based domain adaptation for the scenario where target domain labels are missing. In our experiments, we show that with the domain adaptation network, the accuracy of NYC increases by 4% and the network also achieves the lowest false positives. We also conduct a user study and find that our model outperforms human by 23.12%. In the future, we would like to explore Places trained models instead of ImageNet trained models.

8. Acknowledgments

We would like to thank Precog at IIIT-Delhi for supporting this research work.