Traffic Light Recognition Using Deep Learning and Prior Maps for Autonomous Cars

TL;DR

This paper presents a system that combines deep learning-based traffic light detection with prior maps to improve recognition accuracy for autonomous vehicles, tested on real city routes with successful results.

Contribution

It introduces a novel integration of deep learning detection methods with prior map data for traffic light recognition in autonomous cars.

Findings

Successfully identified relevant traffic lights in city routes

Effective integration of deep learning and prior maps

System demonstrated robustness in real-world scenarios

Abstract

Autonomous terrestrial vehicles must be capable of perceiving traffic lights and recognizing their current states to share the streets with human drivers. Most of the time, human drivers can easily identify the relevant traffic lights. To deal with this issue, a common solution for autonomous cars is to integrate recognition with prior maps. However, additional solution is required for the detection and recognition of the traffic light. Deep learning techniques have showed great performance and power of generalization including traffic related problems. Motivated by the advances in deep learning, some recent works leveraged some state-of-the-art deep detectors to locate (and further recognize) traffic lights from 2D camera images. However, none of them combine the power of the deep learning-based detectors with prior maps to recognize the state of the relevant traffic lights. Based on that, this work proposes to integrate the power of deep learning-based detection with the prior maps used by our car platform IARA (acronym for Intelligent Autonomous Robotic Automobile) to recognize the relevant traffic lights of predefined routes. The process is divided in two phases: an offline phase for map construction and traffic lights annotation; and an online phase for traffic light recognition and identification of the relevant ones. The proposed system was evaluated on five test cases (routes) in the city of Vit\'oria, each case being composed of a video sequence and a prior map with the relevant traffic lights for the route. Results showed that the proposed technique is able to correctly identify the relevant traffic light along the trajectory.

Figures

Tables

Table 1. TABLE I: Distribution of the annotated traffic lights across the datasets’ splits.

Class	DTLD (train)	DTLD (test)	LISA-TLD (train)	LISA-TLD (test)	IARA-TLD
Red	20936	9147	23129	10303	2079
Green	33540	14403	14681	7717	2923

Table 2. TABLE II: Detections Results on DTLD, LISA-TLD and IARA-TLD (%)

	Dataset	Precision	Recall	${\text{AP}}_{50}$
$\tau =0.2$	DTLD (test set)	80.86	91.05	85.62
	LISA-TLD (test set)	62.81	62.49	50.59
	IARA-TLD	60.86	62.28	55.21
$\tau =0.5$	DTLD (test set)	88.59	86.50	85.62
	LISA-TLD (test set)	66.45	54.82	50.59
	IARA-TLD	69.53	57.16	55.21

Table 3. TABLE III: First Correct Detections on Logs

	Log	Delay (seconds)	Distance (meters)
$\tau =0.2$	LR-1	0.56	94.47
	LR-2	0.00	99.70
	LR-3	0.06	99.63
	LR-4 (I)	0.07	98.42
	LR-4 (II)	1.57	84.12
$\tau =0.5$	LR-1	4.50	60.81
	LR-2	0.00	99.78
	LR-3	0.38	96.35
	LR-4 (I)	1.92	78.25
	LR-4 (II)	4.63	58.75

Table 4. TABLE IV: Confusion Matrices for Each Testing Log ( τ = 0.2 𝜏 0.2 \tau=0.2 )

	LR-1				LR-2				LR-3				LR-4				RL
	N	R	G	O	N	R	G	O	N	R	G	O	N	R	G	O	N	R	G	O
NONE (N)	273	0	2	0	432	0	0	0	433	0	0	0	420	0	0	3	2988	0	144	29
RED (R)	0	1294	0	42	0	467	1	0	1	914	0	2	0	296	0	63	10	966	12	37
GREEN (G)	0	0	128	0	1	0	388	0	3	0	156	0	1	0	351	5	10	1	1753	159
OFF (O)	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0

Full text

††publicationid: pubid:

©2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

I Introduction

Autonomous driving is an essential topic of research in the development of intelligent transportation systems. Briefly, the great ambition with autonomous vehicles is fully replacing the human driver by a computer system without compromising and eventually improving safety and efficiency. To this end, the human ability of “seeing” the environment (e.g., the road, pedestrians, signs, and other vehicles) and behaving accordingly should be carefully reproduced by the computer system. In particular, autonomous terrestrial vehicles must be capable of perceiving traffic lights and recognizing their current states (red, yellow, green).

In the autonomous driving literature, the general problem of identifying traffic lights and their states is known as Traffic Light Recognition (TLR). Although TLR is widely addressed in the literature, there are still some challenges to be faced. Most methods [1, 2, 3, 4] focus on locating and/or recognizing all traffic lights in a scene and do not attribute any special meaning to the traffic lights that are relevant for the given context, i.e., the traffic lights in the vehicle’s route and that the driver should obey. Other challenges include recognition in adverse conditions (e.g., rain, snow), early recognition (detecting traffic lights at greater distances), and recognition in different illumination settings (including night images). Despite their importance, these other challenges are out of the scope of this work.

Most of the time, human drivers can easily identify the relevant traffic lights. Nevertheless, there are not always precise rules (e.g., an algorithm) that allow the differentiation of these traffic lights from the others in a scene. To deal with this issue, a common solution is to integrate recognition with prior maps which record position, direction, and other properties of the traffic lights [5, 6, 7]. With prior maps, an autonomous vehicle can be early aware of the presence of traffic lights on its vicinity, and can also fuse the map and real-time sensors’ data (e.g., camera image) for robust location and recognition of the relevant traffic lights in a scene.

Deep learning techniques have showed great performance and power of generalization in many areas and types of problems such as classification [8, 9] and detection [10]. General purpose object detectors have been well explored for traffic related problems (such as detection of pedestrians, traffic signs, etc), and YOLO [11] and Faster R-CNN [12] are two of these state of the art detectors. Motivated by the advances in deep learning, some recent works [13, 14, 15, 16] leveraged some state-of-the-art neural detectors to locate (and further recognize) traffic lights from 2D camera images. Other works, such as [17], combined prior maps with deep learning classification. However, to the best of our knowledge, none of them combine the power of the deep learning-based detectors with prior maps to recognize the state of the relevant traffic lights. Although they are very powerful object detectors, they cannot identify the relevant traffic light for the vehicle, which is a fundamental task for autonomous driving.

Based on this need, this work proposes to integrate the power of deep learning-based detection with the prior maps used by our car platform IARA (acronym for Intelligent Autonomous Robotic Automobile [18]) to recognize the relevant traffic lights of predefined routes. The process is divided in two phases: an offline phase for map construction and traffic lights annotation; and an online phase for traffic light recognition and identification of the relevant ones. In the offline phase, the prior map is constructed by driving IARA on routes of interest to collect camera and LiDAR data. Subsequently, a semi-automatic process is applied to these data in order to find 3D coordinates of traffic light candidates, which are further inspected on camera images to filter false positives and identify those that are relevant for the particular route. The relevant traffic lights’ positions (in 3D world coordinates) are then stored in the prior map. In the online phase, as the car approaches traffic lights, their stored positions are projected to the image in order to filter the predictions (location and state) produced by a deep detector. Finally, only the relevant traffic lights are taken into account in the decision-making process. The relevant state is dictated by the closest traffic light to the projected annotations. It is worth noting that this work treats the traffic light state as a two classes problem (red-yellow and green) in order to cope with the lack of yellow samples for a proper training of a three classes detector. See Figure 1 for an illustration of the system behavior.

For performance assessment, the proposed system was evaluated on five test cases (routes) in the city of Vitória, each case being composed of a video sequence and a prior map with the relevant traffic lights for the route. The performance of the system was measured in two ways: (i) how good were the predicted bounding-boxes, i.e., the performance of the detector itself in terms of the mean Average Precision, and (ii) how accurate were the states predicted by the full system, i.e., the ability of the system to correctly assign the state of the relevant traffic lights during the car progress.

The rest of this paper is organized as follows: related works are discussed in Section II; the proposed method is described in Section III; the experimental methodology is presented in Section IV; results are shown and discussed in Section V; and, finally, Section VI concludes and discusses future work.

II Related Work

This section covers the main works addressing TLR based on prior maps or deep learning applied to TLR. For a more comprehensive review that includes other approaches, the reader should refer to the surveys in [19, 7].

Prior maps containing annotations of traffic lights (e.g., position, direction) can be exploited to increase the robustness of TLR. In this context, the work of Lindner et al. [20] uses maps to recognize traffic lights and they propose a three-stage system: detection (based on handcrafted features), tracking, and state classification. Map information and GPS data can be incorporated into the system to allow triggering the system only near intersections, therefore possibly reducing the amount of false alarm detections. Despite of the use of prior maps, their work does not address explicitly the choice of the relevant traffic lights for the lane.

Fairfield and Urmson [5] present an automatic strategy to map traffic lights by fusing the precise location of the car with image information, and then estimating traffic lights’ 3D positions using least squares triangulation. Levinson et al. [21] propose a mapping procedure of traffic lights using tracking, back-projection, and triangulation. Traffic lights’ states are computed in a probabilistic approach taking into consideration the previously constructed map. In addition to prior maps, Frank et al. [22] locate relevant traffic lights by matching image-based features of intersections. Such features are extracted offline from manually labeled regions around the relevant traffic lights while a neural network is responsible for the final state classification.

John et al. [17] combine prior maps and GPS to limit the region of interest (search area) in which traffic lights are expected to appear. A convolutional neural network (CNN) is used to detect and classify the traffic light candidates within the search area. Jang et al. [23] propose a recognition system that explores the prior maps at every stage. Besides reducing the search area (as in [17]), prior maps include the type of traffic light face that helps state classification. They additionally propose a slope compensation method to treat recognition in inclined roads.

The interest in deep learning methods for TLR has been observed in the recent years. Weber et al.[13] propose a network called DeepTLR for detection and classification of traffic lights. Behrendt et al. [14] modified YOLO [11] to detect traffic light candidates. A custom CNN is leveraged for state classification. A more detailed study on traffic light detection using YOLO can be found in [24]. Pon et al. [16] detect simultaneously traffic lights and traffic signs with a modified version of the Faster R-CNN [12].

Despite the fact that TLR can benefit from modern deep learning techniques, additional information should be used to allow detection/classification of the relevant traffic lights. Therefore, this work proposes to combine a state-of-the-art deep detector with precise localization (e.g., by exploiting LiDAR data) provided in prior maps for a real-world TLR application in autonomous vehicles.

III Traffic Light Recognition for Autonomous Vehicles

This work proposes a method for recognition of relevant traffic lights (TLs) and their state using a deep learning detection model and prior maps of an autonomous vehicle. The process is performed in two steps: offline map construction and traffic light annotation for the relevant lane, and an online detection and recognition of the relevant traffic light state. Figure 2 displays an overall picture of the online phase of the method. The detection model takes camera frames as input and proposes bounding boxes (BBs) with the respective TL state. Subsequently, information from prior maps and the localization of the vehicle is used to project the TL world position to the current frame and then select only one BB to make the final state prediction for that frame. Additionally, a new method for creating prior maps that uses both the detection model and the LiDAR sensor is proposed. The proposed approach is implemented in the autonomous car of our laboratory, IARA.

The next subsections describe: the IARA platform, the autonomous car on which the proposed approach was implemented; the traffic lights detection and recognition procedures, which includes the deep learning detector used to find the traffic lights in the frame and recognize their state; the generation of prior maps used to annotate and store the relevant traffic light world position for a given route; and, finally, the general system operation, that describes how these steps are connected.

III-A Intelligent Autonomous Robotic Automobile (IARA)

IARA is the autonomous car built in our laboratory. It is an adapted Ford Escape Hybrid that features a variety of sensors such as: odometer, LiDAR, IMU, RTK-GPS, and stereo cameras. The LiDAR is a Velodyne’s HDL-32E and the stereo camera is a Point Grey’s Bumblebee XB3 stereo camera. Both are mounted on top of the car, and the camera is mounted front-facing. The HDL-32E LiDAR uses 32 lasers spread at a 40° VFOV (+10° to -30°) to scan 360° horizontally. The Bumblebee XB3 camera has three 1.3MP CCD sensors with 66° of HFOV, and captures colored images of $1280\times 960$ pixels at 16 FPS.

IARA’s software is based on the Carnegie Mellon Robot Navigation Toolkit (CARMEN), which is an open source collection of software for robot control [25]. Our laboratory, LCAD, maintains its own fork of CARMEN at https://github.com/LCAD-UFES/carmen_lcad.

To localize the car accurately in the world, GPS coordinates are often not enough. For this reason IARA uses a localization system [26] based on Particle Filter localization. The system is initialized with a pose from GPS and orientation from IMU, it then enters a cycle of two phases: prediction and correction. In the first phase, the system predicts car poses using odometry data. In the second phase, the method corrects these poses by matching 2D local occupancy grid maps with a global one that is generated offline using the technique presented in [27]. In order to create the 2D grid maps, the localization system transforms 3D point clouds from LiDAR into 2D grid maps. And, to create the global map, IARA applies a SLAM-based system, which uses data from odometer, GPS, IMU, and LiDAR. Experiments showed that IARA’s localization system operates within 0.28 meters of longitudinal error and 0.14 meters of lateral error [26].

III-B Traffic Lights Detection and Recognition

A deep learning neural network model (YOLO [11]) is used for detecting traffic lights and classifying their state. The YOLOv3 (the third version of the YOLO) was chosen because it is one of the state-of-the-art detectors and it can achieve good performances with high frame rate. In this work, only two classes of objects are considered: red-yellow and green traffic light. The choice of mixing red and yellow in one class was to overcome the lack of yellow samples to proper train a three classes detector. Most of the traffic light databases provide very few samples of yellow traffic lights. For the purposes of our application, joining these classes is acceptable since it is better to have IARA stopping before a yellow traffic light than proceeding.

Deep learning detection models usually take a colored image as input for inference. The network forwards the image through the convolutional layers of a standard convolutional neural network (e.g., ResNet, or Darknet) for feature extraction. Further layers process the output of this feature extraction and generate a list of bounding boxes (BBs) with the object’s class probabilities. These are the final outputs of the network.

III-C Prior Maps Generation

The standard mode of operation of IARA is to follow a predetermined path from a Road Definition Data File (RDDF) when driving autonomously. The RDDF is a trajectory that was performed by a human operator driving the car and it stores information like speed, position in the lane, etc. As described in Section III-A, IARA also needs a global occupancy grid map to properly localize itself in the world. To generate this information, IARA has to be driven manually along the route of interest recording all sensor data (referred here as “log”). The sensor data is later used, in a offline process, to build the RDDF and global occupancy grid map.

Given that the recorded log is already necessary for the standard IARA’s operation, it can also be used to create the prior map of relevant traffic lights for the respective RDDF. Making use of prior maps is necessary because until the present moment there is no clear algorithm or machine learning method that can robustly identify which traffic lights are relevant using only image data. The first step is to identify the traffic lights’ positions in the world. For that, the log is played offline, while the traffic light detector locates the traffic lights in the image frames. Additionally, the LiDAR points are projected to camera coordinates, and those that “hit” inside any traffic light bounding box are accumulated in a buffer. The points are projected using the same process as presented in [10]. When eight frames have passed without a single detection, the accumulated point cloud is clustered using the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm [28]. The world position of the cluster centroids are used as traffic lights’ positions. The accumulated point cloud is then reset, and the process repeats throughout the duration of the log. Finally, the centroids are manually filtered to discard false positives and traffic lights that are not relevant for the RDDF of interest. Moreover, traffic lights that share the same control semantics can be grouped together, thus their redundancy can come to use, i.e., when the state of one of them can not be determined, the state of the other is used instead. Traffic lights that are relevant for a particular route are grouped considering a maximum distance threshold (of 20 meters) from each other. One important feature is that the annotated traffic lights can be transferred from one RDDF to the other, for example, when two RDDFs follow the same path (maybe in different lanes). Additionally, the traffic lights that were discarded for not being relevant for one RDDF could be reused for the prior map of another RDDF where they are relevant.

This clustering strategy has to cope with some difficulties: (i) TLs are small objects and LiDAR rays are sparse, thus, in some cases, only few rays will hit a specific TL or perhaps none at all; (ii) small imprecisions in the localization system may cause disturbances in the accumulated point cloud; (iii) if the BB is a little larger than the TL object, then some LiDAR rays may hit inside the BB, but they actually miss the real TL object, hitting something on the background instead. Thus, it is better to train the detector for the offline phase with tight BB annotations, so that BB proposals will also be tight. All of these difficulties, may cause the clustering algorithm to yield false positives and also some inaccurate TL positions. Therefore, they need to be manually filtered as mentioned. In case of any problem in the final accumulated points, there is always the possibility of choosing a single frame for extracting the traffic light world position.

III-D General System Operation

When driving autonomously, IARA continuously checks for the relevant traffic lights associated with the RDDF. Whenever a group of relevant TLs comes within 100 meters, their 3D world position is projected onto camera coordinates. Additionally, the detector is triggered for the current frame. Each annotated TL position is surrounded by a sphere of 1.5 meter of radius that is also projected to serve as a threshold for the localization error. The euclidean distance, in camera’s coordinates, between each BB’s center and each projected TL is calculated. Any BB that has its center outside of all projected spheres are discarded right away. From the remaining ones, that with the closest center to any of the projected TLs is selected, and its status serves as the final prediction for that frame. The final state output of the system is either one of the four: none, when there is no traffic light; off, when there is a traffic light, but the state is not recognized, red, when it is red or yellow; and, green, when it is green. In cases in which all BBs are discarded, the final state is set to “off”. Before a red, yellow or off traffic light, IARA will reduce its speed until a complete stop if necessary, otherwise, before a green traffic light, it will continue its trajectory.

IV Experimental Methodology

The experiments for this work aim mainly to evaluate full TLR system on five driving logs. In addition to the traditional metrics, we also analyzed how early the traffic lights are correctly detected. As preliminary study, we also included the performance evaluation of the deployed detector (an YOLOv3 model). The rest of this section describes the datasets used in the experiments, the driving logs (used to reproduce the IARA’s sensors offline), the performance metrics, and the experiments themselves.

IV-A Datasets for Traffic Light Detection

The training and evaluation of the YOLOv3 detector leveraged two public available datasets: DriveU Traffic Light Dataset (DTLD) [29] and LISA Traffic Light Dataset (LISA-TLD) [7].

The DriveU Traffic Light Dataset (DTLD) comprises traffic image sequences from 11 German cities. Each image ($2048\times 1024$ pixels) includes the following annotations about traffic lights: bounding box coordinates, state, relevance status (visually defined by annotators), horizontal or vertical orientation, occlusion, number of lights, type of traffic light (for pedestrians, cyclists, or cars), and other attributes. For our purposes, pedestrians and cyclists traffic lights were removed. From the remaining, only active (i.e., not “off”) traffic lights which are facing the car and have at least three bulbs were selected. The images are cropped to a $1280\times 960$ rectangle aligned to the top and centered on the horizontal axis, and then scaled down to $640\times 480$ pixels. The bounding box annotations were changed accordingly to reflect the cropping and scaling. The dataset comes with a training-test split.

The LISA Traffic Light Dataset was developed by the Laboratory for Intelligent and Safe Automobiles (LISA) at University of California. The dataset was created using a Point Grey’s Bumblebee XB3 (the same model in IARA) to record more than 44 minutes of traffic video sequences in San Diego, California, USA. Some video sequences were recorded during the day, and others during the night. The dataset is split into training and testing sets, and has seven TL classes: “go”, “go forward”, “go left”, “warning”, “warning left”, “stop”, and “stop left”. The classes “go”, “go forward”, and “go left” were merged as “green”, and the other classes as “red” since we do not address the recognition of yellow traffic lights in separate.

For the full system evaluation, we used a local dataset named (in this work) IARA Traffic Light Dataset (IARA-TLD). It comprises images of traffic scenes recorded in Vitória, Espírito Santo, Brazil with help of the IARA’s camera. The dataset has a total of 5002 bounding box annotations distributed on three different classes: “green”, “yellow”, and “red”. The “red” and “yellow” classes were also grouped into a single class “red”. The images were scaled down to $640\times 480$ pixels. Table I shows the distribution of the classes (i.e., number of annotations per class – “red” and “green”) of the annotated traffic lights across the datasets’ splits.

IV-B Driving Logs

Driving logs are files containing detailed sensory information recorded during the car’s trips, and serve to reproduce the IARA’s sensors offline. These logs can be revisited (playback operation) as many times as needed, which allows to test offline modifications on the IARA’s software, and evaluate the new car’s behavior. A total of 10 logs were recorded on Dante Michelini avenue (Vitória, Espírito Santo, Brazil), from which five were used to construct the prior maps, and the other five for full system evaluation. This avenue was chosen because it contains challenging situations, i.e., several traffic lights where some of them are potentially discordant. There are four bifurcations, each one with three lanes that go straight, and two other lanes (the two leftmost) turning left. In case of turning left, the driver should take one of the two leftmost lanes and obey the corresponding traffic lights. Otherwise, the driver can keep on the three rightmost lanes ruled by the other set of traffic lights. Despite of the current lane, all traffic lights are visible to the camera most of the time. Fig. 1 depicts this ambiguous situation.

The logs (a total of 10) are associated with stretches and lanes in the avenue. The first stretch is showed in Fig. 3. The lanes represented by the blue arrows follow all the avenue. The leftmost (from the car view) is identified by RL, and the other (the middle lane) by RM. The orange arrows are restricted to this stretch, being the leftmost identified by LL-1, and the rightmost by LR-1, where 1 is related to the first stretch. The rest of the LL/LR lanes are numbered according to their corresponding stretch.

IV-C Metrics

The detection performance was measured in terms of precision, recall, and mean average precision (mAP). These metrics were calculated using an IoU (Intersection over Union) threshold of 0.5, and for a confidence threshold ($\tau$) of 0.2 and 0.5.

The mAP calculation follows the definition for the Pascal VOC 2007 competition [30]. Basically, mAP is defined as the mean of the Average Precision (AP) for each class, which, in turn, is the value corresponding to the area under a precision-recall curve. Pascal VOC uses an IoU threshold of 0.5 in the APs’ calculation.

For the full system evaluation, the state of the relevant traffic lights was predicted for all the frames of each test log. We reported the confusion matrix resulting from comparing the predictions with the ground truth annotations. Additional information was recorded (for each stretch/lane) in order to verify how early the system correctly perceived the traffic lights: the time the system took to produce the first correct prediction since the car entered the 100-meters range from the next set of traffic lights; the distance the car is when this occurred.

IV-D Experiments

Two different experiments are performed in this paper. First, a deep learning model (YOLOv3) for detecting traffic lights and their state was trained and evaluated. Second, the proposed system was tested using the driving logs on avenue Dante Michelini.

IV-D1 Training the Detection Model

Currently there are easily available pre-trained models that can detect traffic lights, such as those trained on the COCO dataset [31]. But, usually, these do not identify the traffic light’s state, so we trained our own model. The YOLOv3 [32] was trained for detecting only traffic lights and their state. The model was trained using the training sets of the DTLD and LISA-TLD. The model was validated on the LISA-TLD’s test set using the Pascal mAP metric.

YOLO is a deep learning architecture for object detection, first implemented on the Darknet framework. The third version of this architecture, YOLOv3, uses Darknet-53 (a CNN model with 53 convolutional layers) as its backbone, and reaches 57.9 mAP ($\textrm{AP}_{50}$) on Microsoft’s COCO dataset, using an input resolution of $608\times 608$ pixels. According to YOLOv3’s report, it performs around 3.8 times faster than RetinaNet, which is also a good alternative once its best model achieves 61.1 mAP on COCO. RetinaNet’s slower inference times make us opt for using YOLOv3, for a more real time solution.

The YOLOv3 was trained for 15000 batches, with 64 images per batch, and a constant learning rate of $10^{-4}$. The input resolution was $608\times 608$ pixels as a compromise between inference time and accuracy. Many of the default parameters were also kept, such as: image augmentation with changes in hue, saturation, and exposure; batch normalization; and default anchors. Additionally, YOLO was allowed to change the input resolution from $608\times 608$ to different resolutions following multiples of 32 starting from 320 (e.g., 320, 352, $\dots$, 608) every 10 batches, as it is done in its original work. This, supposedly, makes the model more robust to different scales.

IV-D2 Experiments on Logs

The testing logs on avenue Dante Michelini were used to evaluate the overall system performance. RM was used to build the global occupancy grid-map of the region (Section III-A). RM and LL-{1-4} were utilized for creating prior traffic light maps for the testing logs, which were RL and LR-{1-4}. The detection model outputs many bounding boxes at different confidence values which range from 0 to 1. A confidence threshold $\tau$ is used to eliminate all bounding boxes that have confidence bellow the threshold. By reducing $\tau$, the detector will output more bounding boxes, causing an increase in recall but lowering precision. Since our method uses prior maps to filter false positives, a lower value for $\tau$ could be profitable. Therefore, two values for $\tau$ are investigated, $\tau\in\{0.5,0.2\}$. For each testing log, the final predictions of our system is compared with the ground truth at every frame.

V Results

In this section, the evaluation of our experiments are displayed and discussed. First, the deep detector’s evaluation, then, measurements of the system’s overall performance.

V-A Detection Model

The detection model was evaluated on DTLD, LISA-TLD and IARA-TLD using precision, recall and mAP. A IoU threshold of 0.5 was used, and $\tau\in\{0.2,0.5\}$ for measuring precision and recall, anticipating the values that will be used when evaluating the system. The detector achieves $55.21\%$ of mAP on IARA-TLD and $62.28\%$ of recall at $\tau=0.2$. Results for all the datasets can be seen on Table II. Reducing $\tau$ increases recall, but reduces precision. A higher recall is usually desired when bounding boxes can be further filtered, e.g., using prior maps as in this work.

Taking an image of $608\times 608$ pixels as input, YOLOv3 takes about 47 milliseconds to make proposals on IARA’s video card (Nvidia’s Titan Xp). This is roughly 21 Hz, but this is capped by the low frequency of our camera: 16 Hz.

V-B Entire System

To evaluate the system’s performance, its predictions are compared with ground truth throughout the frames of each test log. Fig. 4 shows this comparison in the format of a timeline. As time progresses, the car approaches different traffic lights. For all test logs, there are some frames without traffic lights (“none”) within a hundred meters. As it can be seen, most of the predictions are consistent with the ground truth, and the inconsistencies usually occur when the car is approaching a traffic light. The first correct predictions are represented on Fig. 4 as yellow downward triangle markers. Table III also reports the first correct predictions on the shorter logs. On this table, it is possible to see the time the system took to first predict the traffic light state correctly, and the distance remaining until the traffic light is reached. For $\tau=0.2$, the first detections occur in average with 0.65 seconds of delay and with a distance of around 93 meters of distance, whereas for $\tau=0.5$ it occur in average with 1.53 seconds of delay and with around 84 meters of distance.

At $\tau=0.5$, the system predicted the traffic light state correctly most of the time, but its predictions were fickle, due to unstable proposals from the detector. The detector is not confident enough at long distances, which indicates that it cannot handle small traffic lights well. One possible solution to increase stability would be to use higher resolution images. If the detector cannot detect the traffic light from the prior map, an “off” state is emitted. These misses at long distances can be seen in Fig. 4, for example, at around 10 seconds on LR-2, and at 6 seconds on LR-4. Reducing $\tau$ to 0.2, increases recall, which in turn, leads to improved predictions at long distances as well as reduces the fickle aspect (e.g., LR-4, from 38 to 46 seconds). As a side effect, reducing the threshold also resulted in more false proposals from the detector along the frames, as expected, but these were not a problem since they are normally filtered out by the use of prior maps most of the time. It is worth noting that a truck passed in front of the traffic light in LR-1 resulting in a occluded traffic light (or off state) that was not annotated, but can be seen in the sequence. Table IV reports the confusion matrices for the system at $\tau=0.2$ so that the reader can have a better idea of the overall prediction performance of the system.

When the car approaches a group of relevant traffic lights, there are two types of error that could occur. The first is predicting the traffic lights are red or off when they are actually green, which represents an inconvenience since it will cause the car to slow down to a stop when it could have proceeded. The second type of error is predicting green traffic lights while facing red ones (or off). This is by far the worst since it could lead the car into the middle of a busy intersection and cause accidents. In our experiments, the second type of error was observed, for brief moments and at a long distance, on the second red traffic light of the RL log (see Fig. 4, between 78 and 118 seconds).

For qualitative analysis, a video for the system running can be seen at https://youtu.be/VhdLpuErJ8E; and, the code and trained model can be found at https://github.com/LCAD-UFES/carmen_lcad/blob/master/src/traffic_light_yolo/README.md.

VI Conclusion

In this paper, we proposed a system for autonomous vehicles that uses deep learning and prior maps for traffic light recognition. First, a deep learning model performs traffic light detection and classification of state in a single step. Subsequently, prior maps are used to select only relevant traffic lights from the proposed detections, filtering out false positives as well. Additionally, a new method for creating prior maps is proposed. LiDAR points are projected to camera; points that hit inside detected bounding boxes are accumulated; and finally this point cloud is clustered to propose traffic lights’ position in world coordinates.

Even though results are promising, much work should be done until such a system can be put on the roads. Particularly, it is important to improve the detector’s performance in order to obtain more reliable results.

As future work, we will investigate the reduction of the search space for traffic lights by proposing ROIs in the input image based on prior maps annotations (which is common in literature). This approach has two main potential advantages: saving processing time by keeping the input image resolution as is, or, alternatively, enabling the use of higher resolution images in order to improve the system’s performance.

Acknowledgment

The authors thank the NVIDIA Corporation for their kind donation of the GPUs used in this research.