TAU: A Framework for Video-Based Traffic Analytics Leveraging Artificial Intelligence and Unmanned Aerial Systems

TL;DR

This paper presents TAU, an AI-integrated framework utilizing UAV video data for automated traffic analysis, including object detection, vehicle tracking, speed estimation, and traffic insights extraction.

Contribution

It introduces novel algorithms for high-resolution UAV image processing, vehicle tracking, and traffic insight extraction, advancing automated traffic analytics from aerial imagery.

Findings

High detection accuracy for small objects in UAV images

Effective vehicle tracking across multiple image crops

Comprehensive traffic insights extraction algorithms

Abstract

Smart traffic engineering and intelligent transportation services are in increasing demand from governmental authorities to optimize traffic performance and thus reduce energy costs, increase the drivers' safety and comfort, ensure traffic laws enforcement, and detect traffic violations. In this paper, we address this challenge, and we leverage the use of Artificial Intelligence (AI) and Unmanned Aerial Vehicles (UAVs) to develop an AI-integrated video analytics framework, called TAU (Traffic Analysis from UAVs), for automated traffic analytics and understanding. Unlike previous works on traffic video analytics, we propose an automated object detection and tracking pipeline from video processing to advanced traffic understanding using high-resolution UAV images. TAU combines six main contributions. First, it proposes a pre-processing algorithm to adapt the high-resolution UAV image as input to the object detector without lowering the resolution. This ensures an excellent detection accuracy from high-quality features, particularly the small size of detected objects from UAV images. Second, it introduces an algorithm for recalibrating the vehicle coordinates to ensure that vehicles are uniquely identified and tracked across the multiple crops of the same frame. Third, it presents a speed calculation algorithm based on accumulating information from successive frames. Fourth, TAU counts the number of vehicles per traffic zone based on the Ray Tracing algorithm. Fifth, TAU has a fully independent algorithm for crossroad arbitration based on the data gathered from the different zones surrounding it. Sixth, TAU introduces a set of algorithms for extracting twenty-four types of insights from the raw data collected. The code is shared here: https://github.com/bilel-bj/TAU. Video demonstrations are provided here: https://youtu.be/wXJV0H7LviU and here: https://youtu.be/kGv0gmtVEbI.

Tables

Table 1. Table 1: Format of the Data stored to record the vehicle movements and statistics over time.

Column Number	Column Name	Description
1	Index	The Primary Key of the table, an Integer value that increments with every new row added inside the table. Every row corresponds to a separate recording.
2	Frame ID	The frame ID already processed in the current recording.
3	Bounding box coordinates	The bounding box coordinates of the detected vehicle considered in this recording. The bounding boxes of all the detected vehicles are generated by the Algorithm 2.
4	Vehicle ID	The tracked ID assigned by DeepSORT to the detected vehicle. The ID is assigned based on the Association metric defined in the Equation 6.
5	Exact time (seconds)	The exact time corresponding to the current frame in seconds.
6	center of the bounding box	The center of the bounding box corresponding to the detected vehicle considered in the current recording.
7	size of the vehicle	The approximative measure of the size of the vehicle: the area of the bounding box.
8	Velocity in km per hours	The instantaneous velocity of the vehicle during the exact time of the current frame.
9	Zone	The zone where the vehicle is located. It will associate the value 0 if it is not inside the user-defined zones. (7 zones are defined in the test).

Table 2. Table 2: Description of 17 of other types of insights that can be extracted from TAU.

N	Insights’ type
8	Number of Vehicles that crossed every line apart from both directions.
9	The vehicle trajectories followed by all the vehicles during the recorded UAV video.
10	The total number of vehicles in all the recorded area over time.
11	The number of vehicles per zone over time.
12	Average velocity of vehicles in all the recorded area over time.
13	Average velocity of vehicles per zone over time.
14	Number of Allowed and Forbidden crosses over time during the recorded session.
15	Correlation degree between the Average velocity and the number of vehicles over time.
16	Factors influencing the number of allowed and forbidden crosses over time
17	Heat map of the maximum recorded velocity per pixel
18	Heat map of the congestion level per pixel
19	Heat map of the average velocity of vehicle per pixel.
20	The diagram of vehicle movements between zones during the recorded UAV video.
21	The pdf (Probability Density Function) of vehicles sizes during the recorded UAV video.
22	The pdf (Probability Density Function) of vehicles velocities during the UAV video.
23	The histogram of vehicle sizes distribution per frame.
24	The histogram of vehicle velocities distribution per frame.

Table 3. Table 3: Performance metrics of the TAU vehicle detection task.

	Yolo v3	Yolo v3 tiny
mAP(0.5)	0.9948	0.7767
mAP(0.5:0.95)	0.8277	0.4451
Recall	0.9965	0.8045
Precision	0.9914	0.8822
F1	0.9939	0.8415

Table 4. Table 4: Pearson correlation coefficients between the values of Average Velocity, the Number of Vehicles and the the Vehicle sizes for the UAV test video .

	Average Velocity	Vehicle size	Number of Vehicles
Average Velocity	\cellcolor[HTML]0364001.	-0.40	\cellcolor[HTML]0099010.70
Vehicle size	-0.40	\cellcolor[HTML]0364001.	-0.39
Number of Vehicles	\cellcolor[HTML]0099010.70	-0.39	\cellcolor[HTML]0364001.

Table 5. Table 5: Pearson correlation coefficients between new forbidden crosses and a set of other factors for the UAV test video .

	Average Velocity	Average size of Vehicles	Number of Vehicles	New Allowed Crosses	New Forbidden Crosses
Average Velocity	1.	-0.50	0.78	0.44	0.30
Average size of Vehicles	-0.50	1.	-0.46	-0.16	-0.32
Number of Vehicles	0.78	-0.46	1.	0.58	0.46
New Allowed Crosses	0.44	-0.16	0.58	1.	0.34
New Forbidden Crosses	0.30	-0.32	0.46	0.34	1.

Table 6. Table 6: Percentage of vehicle moves between the different zones in the crossroad represented in UAV test video .

	Z1	Z2	Z3	Z4	Z5	Z6	Z7
Z1	0	0	0	0	\cellcolor[HTML]FFCE9326	\cellcolor[HTML]FFCE938	0
Z2	0	0	0	0	0	0	0
Z3	0	\cellcolor[HTML]FFCE9317	0	0	\cellcolor[HTML]FFCE9313	\cellcolor[HTML]FFCE936	0
Z4	0	\cellcolor[HTML]FFCE9324	0	0	0	0	0
Z5	0	0	0	0	0	0	0
Z6	0	0	0	0	0	0	0
Z7	0	0	0	0	\cellcolor[HTML]FFCE932	0	0

Equations

$$N\times N\times (3\ast (5+C))$$

$$[z=0],(x,y,z)\in \Re$$

$${\color[rgb]{0.0,0.0,0.0}X=[x,y,a,h,\dot{x},\dot{y},\dot{a},\dot{h}]}$$

$$d^{(1)}(i,j)=(d_j-y_i{)}^T{S}_i^{-1}(d_j-y_i)$$

$$d^{(2)}(i,j)=min\{1-r_j^T{r}_k^{(i)}|r_k^{(i)}\in R_i\}$$

$$c_{i,j}=\lambda d^{(1)}(i,j)+(1-\lambda )d^{(2)}(i,j)$$

$$ET=\frac{FID}{FPS}$$

$$[x_0,y_0,x_1,y_1]$$

$$C(x_c,y_c)=(\frac{x_0+x_1}{2},\frac{y_0+y_1}{2})$$

$$[x_0,y_0,w,h]$$

$$C(x_c,y_c)=(x_0+\frac{w}{2},y_0+\frac{h}{2})$$

$$GS{D}_h=\frac{DAG\ast S_h}{FL\ast I{m}_h}$$

$$GS{D}_w=\frac{DAG\ast S_w}{FL\ast I{m}_w}$$

$$GSA=GS{D}_h\ast GS{D}_w$$

$$A=(x_1-x_0)\ast (y_1-y_0)\ast GSA$$

$$A=w\ast h\ast GSA$$

$$\Re (O,\vec{x},\vec{y},\vec{z})$$

$$\Vert \vec{x}\Vert =\Vert \vec{y}\Vert =\Vert \vec{z}\Vert =1m$$

$$\vec{x}\perp \vec{y}\perp \vec{z}$$

$$P_0:(x,y,z=0)$$

$$\vec{V}=V_x.\vec{x}+V_y.\vec{y}$$

$$P_1:(x,y,z=DAG+FL)$$

$$P_0\Vert P_1$$

$$P_i=M\ast P_w=K\ast [RT]\ast P_w$$

$$P_i=\left[\begin{array}{c}{x}_i\\ y_i\end{array}\right]=K\ast P_w=K\ast \left[\begin{array}{c}{x}_w\\ y_w\end{array}\right]$$

$$\Delta t=(t^{(n+1)}-t^{(n)})=\frac{1}{FPS}$$

$$\{\begin{array}{l}\alpha =(x_c^{(n+1)}-x_c^{(n)})\ast GS{D}_w\\ \beta =(y_c^{(n+1)}-y_c^{(n)})\ast GS{D}_h\\ D=\sqrt{{\alpha }^2+{\beta }^2}\end{array}$$

$$V(m/s)=D\ast FPS$$

$$V(km/h)=3.6\ast D\ast FPS$$

$$Z_i=\{(x_0^{(i)},y_0^{(i)}),(x_1^{(i)},y_1^{(i)}),...,(x_{n_i}^{(i)},y_{n_i}^{(i)})\}$$

Full text

††journal: This is the final proofread version submitted to Elsevier EAAI: please see the published version at: https://doi.org/10.1016/j.engappai.2022.105095

1 INTRODUCTION

In urban areas, people face daily frequent traffic congestion problems. On average, they waste more than 75% of their usual travel time due to congestion [23]. The situation will be more and more complicated by the increased number of people moving to live in cities. The UN world Urbanization prospects estimated in its 2018 report [56] that around 2.5 billion people will be living in urban areas by 2050. Hence, the traffic congestion problems are going to be increasingly aggravated. To analyze and monitor the traffic jam, surveillance cameras are placed in the critical parts of the city roads [30]. Although being informative, these cameras suffer from limited and incomplete vision areas. They can not cover the whole portion of the road in congestion. Moreover, the high cost of installing and maintaining these cameras is also a considerable limitation. Thus, Unmanned Aerial Vehicles (UAVs) are an attractive tool for analyzing traffic. This is why the authorities are increasingly adopting them all over the world. This increasing adoption is due to many factors. First, they are flexible and could be rapidly flown over the congestive portion of the roads to take a real-time video of the traffic area. Second, they need no additional installation or maintenance cost for this analysis mission. Third, they could easily be connected to 5G networks to stream the captured videos in high bandwidth while maintaining high resolution. Hence, the UAVs possess a high potential to be used for various tasks related to traffic management and analysis.

The advantages of using UAVs does not imply giving up the use of stationary cameras. There are many scenarios in which stationary cameras are more appropriate than UAVs. Indeed, UAVs suffer from battery constraints that hamper their adoption for continual use. Also, Ensuring stable temporal coverage by using only UAVs is challenging. Moreover, bad weather conditions may complicate their operation. Besides, the safety and the compliance with different civil aviation laws is another limitation to face. Hence, it is better to combine stationary cameras and UAVs to overcome the limitations of both tools. This improves the robustness of traffic analysis and management. However, before studying the possibility of this combination, this paper aims to profoundly study the potential of using only UAVs in this task.

Currently, most use cases for the UAVs inside the roads are deprived of machine intelligence. The majority of the efforts are focused on gathering real-time videos of the traffic for visual analysis and investigation by an engineer. The engineer will train himself to recognize the causes of traffic violations or how to choose suitable measures to reduce congestion. He will decide on more challenging problems like how to optimize the traffic signals and the time slots allocated for every road portion. More critical situations can face the engineer, such as the accident risk assessment. He will have to report to the auto insurance industry and the police departments to help them judge the potential responsibility of the counterparties. These are some of the challenging missions that create an insistent need for an automatic tool that alleviates the process and makes the monitoring and the management more efficient and less prone to errors.

On the other hand, Since 2012 [43], an impressive advancement in computer vision applications has been noted. This advancement is due to the appearance of the area of deep learning algorithms that made image and video processing more efficient than ever before [28, 46, 10, 9, 12, 3, 4, 42, 15, 5, 7, 11, 41, 48, 8]. Many deep learning algorithms have been incrementally optimized for the main tasks on deep learning such as object detection inside images [46], object tracking in videos [22], instance segmentation [31] and semantic segmentation [32].

These advancements in deep learning have inspired a variety of research projects that aimed at maximizing the use of UAV images to improve traffic services [49]. However, the variety of studies and techniques hardens the mission for a transportation engineer to pick out the best technique for every insight he wants to extract from the video. Moreover, he will face difficulties in testing the feasibility of every technique apart. Besides, integrating the selected techniques will be another issue to resolve.

This paper aims to target this matter and to design a unified and straightforward pipeline for the extraction of a comprehensive set of insights from the UAV video (more than 20 types of insights). The founding Framework for this pipeline is coined TAU (Traffic Analysis from UAVs). It helps the Transportation Engineer to automatically analyze the traffic from a UAV video in a seamless way. To summarize, TAU introduces six main contributions:

The first contribution is a new pre-processing algorithm to enable the object detector to work in parallel on high-resolution images. The algorithm decomposes the input frame into multiple small crops to keep the ability to detect the small-sized vehicles.

• 2.

The second contribution is a new algorithm to recalibrate the vehicle coordinates so that the multi-object tracker can efficiently identify all the circulating vehicles in the high-resolution video.

• 3.

The third contribution made is a new algorithm to calculate every vehicle’s speed in the video. The mathematical formulation behind this algorithm is explicitly detailed.

• 4.

The fourth contribution is a new algorithm to calculate the number of vehicles per traffic zones. The algorithm follows the Ray Tracing approach in treating this problem.

• 5.

The fifth contribution is a new crossroad arbitration algorithm. This algorithm is based on the data gathered from the different traffic zones surrounding it.

• 6.

And finally, the sixth contribution is a set of algorithms for extracting twenty-four types of insights from the traffic video. These insights enable better traffic understanding and analysis using histograms, heatmaps, curves, and animations. The TAU framework was a successful research prototype designed in an industrial context and opened later to the community.

This paper is structured as follows: Section 2 will describe the research works related to the use of UAVs inside the domain of traffic analysis and management. Then, section 3 is reserved for the theoretical founding of the TAU framework. Later, Section 4 describes the implementation details made to test TAU as well as the generated results. Finally, section 5 concludes the utility of TAU and describes the limitations to address in future works.

2 Related Works

This section inspects the research works that used UAV imagery to enhance traffic monitoring and management. The majority of these research works are focused on three classes [38]. The first class is road detection; the second class is vehicle detection and tracking; the third class is traffic parameter estimation. A description of the research works implemented in every class will be detailed in the following subsections.

2.1 Road detection

Road detection from UAV was the target of many studies in literature [27, 63, 45, 40, 50]. The aim was to localize the areas where the different types of vehicles circulate. This goal is essential to automatically make the UAV focus on the areas of interest and neglect other regions. In some cases, using the geo-coordinates got from the UAV helps to localize the boundaries of the road detected by the camera. Especially when the camera’s view is orthogonal to the road itself and without any inclination. However, in many cases, these geo-coordinates could not be captured accurately for every frame of the UAV video. Moreover, these geo-coordinates become useless when the camera is inclined. In this case, deducing the correct location of the road boundaries becomes impossible without considering the images captured from the UAV. Hence, it is recommended to use computer vision for the generic approaches of road detection. For example, Zhou et al. [63] were among the first to design an algorithm for both the detection and the tracking of roads. Moreover, the method was efficient when tested on multiple scenarios. Kim et al. [40] proposed another algorithm that is trained on the road structure using one video frame. The algorithm then predicts road locations in the remaining frames of the video.

2.2 Vehicle Detection and Tracking

Vehicle detection and Tracking from UAV was the main target for most research works that leveraged the UAV imagery for traffic analysis [33, 21, 20, 61, 17, 25, 34, 19, 18, 60, 58, 2, 29]. In addition, detecting and tracking vehicles were the primary goals since the adoption of UAVs for civil use. Vehicle Detection is considered the first building block for any framework that analyzes traffic. Similarly, Vehicle Tracking is considered the second building block for this kind of framework. In the work introduced by Cao et al. [19], a system of Vehicle Detection and Tracking was designed based on multi-motion layer analysis in an unsupervised way. It deduces the vehicle position by subtracting the foreground from the background. However, later, supervised algorithms were increasingly adopted due to their efficiency. Supervision given by the user enables learning the circulating vehicles’ patterns using a structured dataset of UAV images. SVM and CNNs have shown significantly better efficiency in detecting vehicles [20, 62]. The state-of-the-art object detectors were used to detect vehicles similarly to other objects.

2.3 Traffic parameters estimation

Traffic parameters estimation from images captured using a UAV was the target of numerous research studies [47, 6, 59, 55, 54, 36, 37, 62, 24]. Their estimation could be performed offline or online, following the computation needs and the implementation constraints. Many traffic parameters were targeted in literature: density of vehicles inside the roads, speed, travel time, traffic congestion, delay due to congestion, annual average daily traffic. For example, McCord et al. [47] were among the firsts to introduce a method to estimate the annual average daily traffic from aerial imagery. Shastry et al. [54] tried to estimate a subset of traffic parameters based on motion information and image registration. More recently, Ke et al. [37] designed a method to measure the speed, the vehicle density, and the volume of the traffic from aerial imagery. However, motion-based vehicle detection affects the method efficiency in congestive areas. Ke et al. [38] tried to solve this limitation by introducing a framework that works for both congested and non-congested traffic conditions. Gattuso et al. [26] studied the use of drones to extract traffic parameters such as: vehicle classification, estimation of the traffic flow, the traffic density, and the speed in contexts related to urban roads and freight terminals. However, their work does not represent a solid framework for precise insights from traffic data. It represents an initial effort in this direction. Li et al. [44] targeted the problem of traffic analysis at nighttime. They proposed a domain adaptation algorithm to maximize vehicle detection accuracy at nighttime. Their method did not need a labeled dataset for nighttime images, and it is trained only on daytime images. Their work aims to introduce a generic method for traffic parameter analysis that works on both daytime and nighttime. Ke et al. [35] made a framework for traffic parameters estimation from moving UAV video. Although their work is interesting, the challenging problem of moving cameras makes their solution not yet mature enough to be used directly by a traffic engineer. This study aims to ensure the precision of the metrics by specifying some conditions in the UAV camera view itself.

Brahimi et al. [16] investigated the potential of the drones for traffic parameters estimation. They took as an example an urban roundabout. Then, they prove that drones are better than conventional sensors for estimating traffic parameters in this case. The study was limited and did not present a clear framework for extracting precise insights from the drone video. They intended to point out the efficiency of drones compared to conventional sensors.

In 2016, Khan et al. [39] made a review about the research works related to traffic analysis from UAVs. Based on it, they introduced a Framework that summarizes the technologies needed for the different steps of analyzing traffic from UAV videos. This Framework is divided into seven steps or components: (1) definition of the scope, (2) planning of the UAV flight, (3) implementation of the flight, (4) acquisition of the data, (5) processing and analysis of the data, (6) interpretation of the data, and finally (7) application for an optimized traffic. Although being valuable, their work has some limitations. First, it does not present a clear Framework for straightforward and precise extraction of measurable insights from the captured UAV video. Second, it is not focused on the UAV video analysis itself, which is the main challenging component. This component is treated in a shallow manner equally to far simpler components like the definition of the scope, for example. The reader does not have any clear insight he can precisely have from the UAV video. Moreover, the work is a bit outdated and needs to be refreshed, especially in the image analysis domain. This domain has known a considerable change in the past five years.

From the above study, we can deduce the diversity of the research works that targeted traffic analysis from UAVs. However, to our knowledge, no one gave the community a clear and unified pipeline for the extraction of a comprehensive set of traffic data from one UAV video. Consequently, the current study is targeting this gap. It introduces a practical and straightforward pipeline for understanding the traffic from UAV video. It provides the theoretical basis and the implementation details to be followed. The source code is opened to the community. The Framework of this pipeline is coined as TAU (Traffic Analysis from UAVs). The next section (Section 3) demonstrates the theory behind TAU before validating the implementation in Section 4.

3 TAU: The Proposed Method

3.1 Method Overview

The TAU framework is based on two main building blocks: Vehicle Detection and Vehicle Tracking. For the first building block, the vehicle detection task, it operates on the UAV video frame by frame to detect all the occurrences of vehicles inside one frame. To this end, we built a specific vehicle detection dataset on UAV images. The annotated vehicles inside the dataset were grouped into five classes (Car, Bus, Truck, Bicycle/Motorcycle, Pedestrian). Then, we trained one state of the art object detector (YOLO v3 [53]) on this dataset to detect vehicles inside the UAV video.

Concerning the choice of YOLO v3, it can be replaced by another object detector: either a generic object detector or a dedicated object detector for aerial imagery. For generic object detectors, we can choose YOLO v4 [14] which is a more recent version of the YOLO family released in April 2020. It comes with many improvements over YOLO v3. However, we used YOLO v3 in TAU because, at the time of the implementation of the code, YOLO v4 was not released. The replacement of YOLO v3 by YOLO v4 is one of the improvements we listed for the next version of TAU (see Section V). Moreover, a dedicated object detector for aerial imagery can be used. One of the most recent works in this field is the Butterfly Detector [1].

Although the model is trained to detect pedestrians, this class is omitted during the study to focus only on vehicles in the current stage. Later, if there is a need to analyze the pedestrian movements, the same model can be customized. The trained model extracts for every frame the bounding boxes around every vehicle. This information is saved to use later by the second building block.

Concerning the second building block, the vehicle tracking task, it takes as input the detected vehicles in the current frame and tries to associate every one of them to their correct ID. There are two cases for this association. In the first case, if one of the detected vehicles has not been seen before in the previous frames, the tracking algorithm associates to it a new identifier. In the second case, if this newly detected vehicle existed already in one of the previous frames and got an identifier $X$, the tracking algorithm associates this vehicle to this identifier. After that, all the bounding boxes related to one vehicle during the video are saved in a separate dataset. These data help generate the statistics related to the vehicle movement over the whole video. Concerning the tracking algorithm selected for TAU, we picked out one state of the art algorithm in the domain of Real-Time Multi-Object Tracking: DeepSORT [57].

In the following subsections, we will discuss the theoretical basis of the two building blocks of TAU (Subsections 3.2 and 3.3). Next, the theoretical basis of the different insights will be discussed based on the dataset generated at the end of the two building blocks.

3.2 Vehicle detection using YOLO v3

3.2.1 YOLOv3 overview

Introduced in 2018, YOLO v3 [53] is an incremental improvement made over the previous versions: YOLO v1 [51] introduced in 2016 and YOLO v2 [52] in late 2016. It is a popular object detection algorithm, especially for real-time applications due to its efficient processing time compared to competitors [53].

The YOLO v3 detection process is clarified more in $Figure$ 1. First, the input image is resized to $416*416$. The resulting image is divided into a grid of $13*13$ cells; every cell is of size $32*32$ pixels. In YOLO v3, the number of the grids depends on the size of the input image. In the current case, the input image given to the network is of size $416*416$, which can be divided into a grid of $13*13$ cells, every cell is of size $32*32$. If it was $608*608$, it will be divided into a $19*19$ grid, every cell is always of size $32*32$. After the partitioning step, the model predicts the class probability and the bounding boxes for every grid cell with a confidence score. This information is assembled to deduce the final detected object in the input image (most relevant bounding box and the class it belongs to).

The model is trained on six classes: class Car, class Bus, class Truck, class Motorcycle or Bicycle, and finally the class Pedestrian (this class is omitted in this study). YOLO v3 does the whole prediction using only one convolutional network. It takes as input the resized image and predicts as output a tensor of dimension

[TABLE]

where:

$N\times N$: is the number of grid cells.

• 2.

$C$: is the number of targeted classes.

• 3.

$(3*(5+C))$: This is because 5 parameters are detected ($t_{x},t_{y},t_{w},t_{h}$, and the box confidence score) plus the probability for every one of the classes ($C$). This is done for every grid cell at 3 different scales.

In the trained model, this tensor will be of size $13*13*(3*(5+6)$, which corresponds to a 3D tensor of dimension $13*13*33$. The main architecture of YOLO v3 network is displayed in $Figure$ 2.

The UAV collected the images from an altitude of 175 meters. This altitude is chosen to maximize the surveyed area. This gives a wide perception of a crossroad intersection. Including a clear view of the roads carrying vehicles inside and outside the intersection. The video is captured in 4K resolution ($4096*2160$ pixels). High resolutions like 4K are recommended for better analysis of the traffic. Lower resolutions could lead to blurred views or lower accuracy.

3.2.2 Dealing with High-Resolution Images

Although high resolutions are recommended for the input video, they cannot be used directly in the training of YOLO v3. This limitation is due to the heavy computational load. The maximum resolution that can be used at a reasonable computation cost is $608*608$. For the construction of the training dataset, a perpendicular view on the ground should always be kept. Therefore, only UAV images that are perpendicular to the ground are considered in this study. This constraint simplifies one dimension in the mathematical equations used during the analytics. For the equations used in the insights’ extraction, the following hypothesis is assumed: The vehicles are circulating on the plane of the equation:

[TABLE]

Where $(z)$ refers to the third dimension in the 3D coordinate system $\Re(x,y,z)$. More details about the mathematical formulation are given in section 3.4

Finally, the dataset is constructed by collecting croppings of size $512*512$ from different locations inside the captured UAV images, but croppings that contain vehicles are preferred. After that, the annotation of the different classes is made.

During the framework test on a sample UAV video of the traffic, the view of the test video should be perpendicular to the ground. During the execution of the TAU framework, the test UAV video is processed frame by frame. Every frame dimension is extended to the nearest integer dividable by $512$, as shown in Algorithm 1.

After preparing the pre-processed frame, it is divided into a grid of processed cells. Every processed cell is of size $512*512$ pixels. Each of them is then passed to the YOLO v3 trained model to detect the vehicles inside it. Then, the resulting local coordinates for every vehicle are scaled accordingly to deduce their coordinates in the global frame. This task is done following the Algorithm 2.

In the end, the final list of the detected vehicles are passed as input for the tracking algorithm to decide for every vehicle if it belongs to an already tracked vehicle, or it is a newly intervening car inside the scene (as detailed in Section 3.3).

3.3 Vehicle tracking using Deep SORT algorithm

To track the detected vehicles, the DeepSORT algorithm [57] is chosen. It is a state-of-the-art online multi-object tracking algorithm [22]. It was an improvement over its earlier version: the SORT [13] (Simple Online and Real-time Tracking) algorithm. DeepSORT has an accuracy comparable to the state-of-the-art trackers but with a better processing time (up to 60 frames per second).

In TAU, DeepSORT processes the UAV video frame by frame. For every frame, it takes the detected vehicles generated by the Algorithm 2. Then, it tries to associate them to the previously tracked vehicle ID or a new tracked vehicle ID. Finally, the motion model state of the vehicle is implemented using eight parameters:

[TABLE]

where $x$ and $y$ are respectively the horizontal and the vertical coordinates of the vehicle location in the image, which is associated to the center of its bounding box, $a$ is the aspect ratio of the vehicle bounding box, $h$ is the height of the bounding box, and the other parameters are their respective estimated velocities in the image domain. Kalman filter is used to predict the new motion state of every vehicle based on the previous measurements.

If a detected vehicle is associated to an ID, the bounding box parameters are used to update the target motion model state parameters using a Kalman filter. The association of the newly detected objects to the already tracked targets is made via calculating the IoU (Intersection Over Union) between every detection bounding box and all the Kalman-filter predicted bounding boxes of the existing vehicle IDs. If the IoU is lesser than a fixed value $IoU_{Min}$, the assignment is rejected and the detected object is assigned to a new target. The assignment is implemented via the Hungarian algorithm.

Deep SORT [57] added the appearance information into the association operation. A designed Convolutional Neural Network (CNN) architecture generates a deep appearance descriptor for every detected vehicle. The CNN architecture is trained offline on a large re-identification dataset.The CNN architecture is illustrated in $Figure$ 3. The inclusion of the appearance descriptor in DeepSORT improved the algorithm’s ability to deal with longer periods of occlusions without affecting the high frame rate during the processing. This descriptor has also reduced the number of identity switches. Besides, it made the algorithm robust to the possible overlaps between the bounding boxes of different tracked vehicles.

Two metrics are combined to associate the new detected vehicles to the tracked targets. The first metric is the motion metric. It is expressed as the squared Mahalanobis distance between the detected object motion state and the predicted motion state for every tracked target already saved. The squared Mahalanobis distance is defined in $Equation$ 4:

[TABLE]

where $d_{j}$ is the $j^{th}$ detected bounding box, $(y_{i},S_{i})$ is the projection of the $i^{th}$ track distribution into the measurement space.

The second metric is the smallest cosine distance between the appearance descriptor of the detected object $r_{j}$ and the previous appearance descriptors of the $i^{th}$ tracked vehicle ID in the previous frames . This metric is expressed in $Equation$ 5:

[TABLE]

where $R_{i}$ is the list of the last appearance descriptors for the $i^{th}$ tracked vehicle, $R_{i}=\{r^{(i)}_{k}\}_{k=1}^{L_{k}}$, $k$ is the index of the appearance descriptor belonging to this list $R_{i}$. This $k$ is an integer that increments from 1 to a limit value $L_{k}$ that is a constant set by the user during the implementation. In our case, $L_{k}$ took the value 100 during the experiments.

The two metrics are combined using a weighted sum:

[TABLE]

The $\lambda$ parameter controls the weight of every metric inside the association.

After the association step, every detected vehicle inside the current frame will be associated with a vehicle ID. If the vehicle is already present in the previous frames and got a specific ID, it will be assigned to this ID. If the vehicle is newly seen inside the scene, it will be assigned to a new ID. This ID is pivotal information in the TAU framework. Most statistics calculated in the next subsection are based on it (please see subsection 3.4).

3.4 Extraction of insights from the UAV video

After processing the frame, the system generates a list of the detected vehicles inside this frame. Two major pieces of information characterize every vehicle. The first is the bounding box information (Generated by Algorithm 2). The second is the tracked ID of the vehicle; this is assigned by Equation 6. For every processed frame ID, a separate database of statistics is needed to analyze the movement of the vehicles. The format of the relational table used to record these statistics is described in $Table$ 1.

The considered relational table contains nine columns. The first column is the index, an integer value that increments with every recording added to the table. Then, we have the three primary columns already described above: the Frame ID, the Bounding box coordinates, and the Vehicle ID. From these three columns, we automatically deduce the remaining five columns. These five columns can be calculated online during the processing of the frame or offline after the processing of the total UAV video. They are not extracted directly from the video like the three main column values.

Hence, at the end of processing one frame, the data related to every vehicle is saved in separate rows. Every row corresponds to the information of every vehicle detected in the processed frame.

From the saved database, a set of traffic information could be extracted. The theoretical basis for every extracted information is detailed inside the following paragraphs.

3.4.1 Exact time of the frame

This information can be directly deduced from the Column 2 (Frame ID) by using the following Equation:

[TABLE]

Where $ET$ is the Exact time expressed in seconds. $FID$ is the Frame ID. It is an integer value initialized to zero before processing the video. It is incremented after the processing of every frame apart. $FPS$ is the frame per second information extracted directly from the UAV video.

3.4.2 Center of the bounding box of the vehicle

This information can be extracted directly from Column 3 (Bounding box coordinates of the Vehicle). The Bounding box can have two possible formats. The first format is:

[TABLE]

Where $(x_{0},y_{0})$ are the coordinates of the upper left corner of the Bounding Box, $(x_{1},y_{1})$ are the coordinates of the lower right corner of the Bounding box. For this format, the center point is deduced using the following Equation:

[TABLE]

The second format for the bounding box is:

[TABLE]

Where $w$ and $h$ are respectively the width and the height of the bounding box. For this format, the center point is deduced using the following Equation:

[TABLE]

3.4.3 Approximate size of the vehicle

Following the Hypothesis defined in the Equation 2, the size of the top-down view of the vehicle can be approximated to the Area of the bounding box. Since the bounding box area is expressed in pixels, we need to convert it to the metric scale using the UAV image’s Ground Sample Distance $GSD$. $GSD$ is expressed using the following Equations:

[TABLE]

[TABLE]

where $DAG$ is the distance above the ground, it corresponds to the height between the lens of the UAV camera and the ground. $S_{h}$ and $S_{w}$ are respectively the height and the width of the camera sensor itself. $FL$ is the focal length of the camera, which is the distance between the sensor of the camera and its lens. All of $DAG,S_{h},S_{w}$ and $FL$ are expressed in meters. $Im_{h}$ and $Im_{w}$ are the width and the height of the whole captured image in pixels. Finally $GSD_{h}$ and $GSD_{w}$ are expressed in meter per pixel $m/px$.

From $GSD_{h}$ and $GSD_{w}$, we can deduce the Ground Sample Area $GSA$, which is the area associated with every pixel in the UAV image. It is expressed in squared meter per pixel $m^{2}/px$ and calculated using the following Equation:

[TABLE]

Based on $GSA$, we can calculate the size of the vehicle $A$, which is approximately the area of the Bounding box. $A$ is expressed in squared meters $m^{2}$ and calculated using the following Equation:

[TABLE]

This Equation is valid for the format defined in the Equation: 8. However, for the second format defined in the Equation 10, It is expressed as:

[TABLE]

3.4.4 Vehicle Velocity

During the video processing, the instant velocity of the tracked vehicle is updated for every frame. It is initialized to $0\ km/h$ for the first frame. Then, it is updated by deducing it from the database referred in $Table$ 1. However, five hypotheses are assumed to simplify the velocity estimation based on the constraints predefined in this study.

First, we consider the Cartesian coordinate system:

[TABLE]

$O$ is the origin of the coordinate system located on the physical point on the ground surface projected on the upper left corner of the UAV image. This point is fixed during all the processed UAV video. $\vec{x}$, $\vec{y}$ and $\vec{z}$ are the unitary vectors of this coordinate system:

[TABLE]

They are also orthogonal to each other:

[TABLE]

$\vec{x}$ is oriented from the origin $O$ to the physical point on the ground surface projected on the upper right corner of the UAV image. Similarly, $\vec{y}$ is oriented from the origin $O$ to the physical point on the ground surface projected on the lower-left corner of the UAV image. $\vec{z}$ is orthogonal to the plane $(\vec{x},\vec{y})$ and oriented from the origin $O$ following the right-hand rule.

Second, we assume that the physical ground surface is approximately a part of the hyperplane $P_{0}$ of $\Re$ defined by:

[TABLE]

This hypothesis engenders that the z component of the vehicle velocity is null. Hence, the velocity of any vehicle on UAV video can be expressed as:

[TABLE]

Third, we assume that the camera view is perpendicular to the ground surface following the same direction of $\vec{z}$. Hence, the surface of the camera sensor is a part of the hyperplane $P_{1}$ defined by:

[TABLE]

Thus, we deduce that the hyperplanes $P_{0}$ and $P_{1}$ are parallel:

[TABLE]

Now, we model the camera following the standard pinhole camera model. The camera will have its separate coordinate system $\Re(C,\vec{i},\vec{j},\vec{k})$, where $C$ is the center of the camera, which corresponds to the pinhole itself. $\vec{i}$ is parallel to $\vec{x}$ and have the same norm and distance. Also, $\vec{j}$ is parallel to $\vec{y}$ and have the same norm and distance. $\vec{k}$ is perpendicular to the sensor and points toward it. Every point $P_{i}$ seen on the image is a projection of a real world point $P_{w}$ that exists on the ground. The projection follows the camera matrix model presented by the equation 24:

[TABLE]

where $M$ is the full projection matrix of the camera. $M$ combines both the intrinsic and extrinsic parameters. The intrinsic parameters are identified by the matrix $K$. The extrinsic parameters are represented the matrix $[R\ \ T]$. $R$ and $T$ are the rotation and the translation matrices between the world coordinate system $\Re(O,\vec{x},\vec{y},\vec{z})$ and the camera coordinate system $\Re(C,\vec{i},\vec{j},\vec{k})$.

Based on the previous hypotheses, the rotation matrix $R$ is null. Also, the translation matrix has only one component over the axis $\vec{z}$. Based on the equation 20, all the components over $\vec{z}$ can be neglected and we can work only on $\vec{x}$ and $\vec{y}$. Thus, we can omit the extrinsic parameters of the camera $[R\ \ T]$ and keep only the intrinsic parameters $K$. After calibration of the camera, these intrinsic parameters are identified which permits to easily have the projection coordinates of every seen point on the ground from the camera perspective without need of conversion between the coordinate systems. Hence, we can assume for our case the following camera projection model identified in the equation 25 below:

[TABLE]

After that, it remains only to convert the projected image from the pixel scale into the metric scale using the Equations 12, 13 and 14. Hence, we can follow our calculations on the 2D representation of the vehicle displayed on the UAV image converted to the metric scale. It is an exact camera projection of the corresponding vehicle on the ground surface belonging to the hyperplane $P_{0}$ (Equation 20). Moreover, based on the Equation 21, we can deduce that the Velocity $\vec{V}$ of a vehicle is the same as the velocity of the 2D projection of the vehicle displayed on the UAV image converted to the metric scale.

The fourth hypothesis to consider is that the inertial point needed to estimate the global velocity of the vehicle can be approximated as the center point of the 2D bounding box of the vehicle. This approximation is based on the previous equations and deductions.

The fifth hypothesis to consider is that the movement of one vehicle from one point $C_{(n)}(x_{c}^{(n)},y_{c}^{(n)})$ (the center point of the bounding of the vehicle in the frame $(n)$) to the point $C_{(n+1)}((x_{c}^{(n+1)},y_{c}^{(n+1)}))$ (the center point of the bounding of the vehicle in the frame $(n+1)$) can be approximated as a linear movement from $C_{(n)}$ to $C_{(n+1)}$. This allows to Approximate the instant velocity between these two frames as the Euclidian distance between $C_{(n)}$ an $C_{(n+1)}$ divided by the time $\Delta t=$ needed for this movement. This time corresponds to the time between two frames, which is:

[TABLE]

where $FPS$ is the frame rate of the UAV video, $\Delta t$ is expressed in seconds.

The Euclidian distance $D$ between $C_{(n)}$ and $C_{(n+1)}$ can be expressed as:

[TABLE]

where $\alpha$ is the movement of the vehicle following the axis $\vec{x}$ and $\beta$ is the movement of the vehicle following the axis $\vec{y}$. All of $\alpha$, $\beta$ and $D$ are estimated in the metric scale. From all the equations above, the velocity of one vehicle $V$ can be expressed in the metric scale $m/s$ or $km/h$ using the following Equations:

[TABLE]

[TABLE]

3.4.5 Number of vehicles per zone

To give better analytics on the different portions of the roads, the user can freely define a set of zones to generate statistics for every zone apart. A zone is a closed polygon composed of points in a specific order. Every closed polygon starts from one point and ends by the same point forming a cycle. Every two successive points are connected using a segment. The set of points is named the vertices of the polygon. The list of segments is named the edges of the polygon. We consider $p$ the set of zones defined by the user. By default, $Z_{0}$ is considered the default zone outside the zones defined by the user. For $(i=1..p)$, Every defined zone by the user $Z_{i}$ can be formulated as:

[TABLE]

$Z_{i}$ is a closed polygon, which has the following constraints:

[TABLE]

For every frame, every vehicle is assigned to the correct zone: user-defined zones $\{Z_{i},i=1..p\}$ or the default zone $Z_{0}$. To be able to do this assignment, we adopt the approximation that if the center point $(x_{c},y_{c})$ of the vehicle is inside one user-defined zone $Z_{i}$, it will be assigned to it. Otherwise, it will be assigned to zone $Z_{0}$.

The problem of checking whether $(x_{c},y_{c})$ is inside the closed polygon $Z_{i}$ is named in computational geometry the PIP problem, PIP means Point In Polygon. The Ray Casting algorithm is among the most used algorithms to solve this problem. We adopted it for counting vehicles inside the zones. The Ray Casting is based on two steps. The first step is to draw a ray that starts from $(x_{c},y_{c})$ and follows any direction. The second step is to count the number of times this ray intersects with the polygon edges $Z_{i}$. If this number is odd, then $(x_{c},y_{c})$ is inside $Z_{i}$. Otherwise, it is outside it. For every frame, we loop over all the tracked vehicles to associate every one of them to the right zone. The process is formulated in Algorithm 3.

3.4.6 Other extracted types of insights

From the above, TAU is based on seven essential types of insights extracted from the UAV video: the Bounding Box Coordinates, The Vehicle ID, the Exact time of the frame, the Center of the Bounding Box, the size of the vehicle, the Velocity, and the Zone. In this section, the mathematical formulation of these types of insights is deeply detailed. Based on them, the Transport Engineer can extract many other types of insights to analyze the traffic information efficiently. In the Experimental results section (Section 4), TAU is applied to generate 17 other types of insights. Their utility for better understanding of the traffic will also be discussed. The Transport Engineer can reuse them or apply TAU for extraction of other types of insights. The source code used to extract these 17 types of insights is shared with the community111The source code is planned to be shared after the publication of the paper..

4 Experimental results

In this section, TAU is applied to extract 17 other types of insights. They are enumerated in Table 2. The enumeration starts with the number $8$. In fact, the first 7 are the main types of insights in TAU and they were described above.

The types of insights extracted from TAU are not limited to the types described in Table 2. Much more insights can be extracted following the needs of the Transport Engineer.

In the following subsections, we will first measure the accuracy of the vehicle detection task in TAU. This task is the base for all insights concluded later. Then, we will show an example of extraction of the seven main types of insights in TAU. After that, we will show, one by one, the result of extraction of the other 17 types of insights described in Table 2.

4.1 Vehicle Detection Accuracy in TAU

To estimate the correctness of the TAU statistics, we should first assess the accuracy of the vehicle detection task. Thus, we constructed an object detection dataset and labeled five types of classes (car, bicycle/motorcycle, bus, truck and pedestrian). Later, the class pedestrian is kept for future studies and the bounding boxes related to this class are omitted from the final visualizations. After the dataset construction, we trained YOLO v3 [53] on it.

To measure the Vehicle detection accuracy, we used the precision and recall metrics, which are defined as follows:

[TABLE]

[TABLE]

where $TP$ is the true positives (correctly detected vehicles), $FP$ is the false positives (false detection of vehicle) and $FN$ is the false negatives ( vehicles not detected).

Another metric is used to estimate the global performance of the vehicle detection task. This metric is the $F1$ metric, which is a combination between the recall and precision metrics and defined in the following equation:

[TABLE]

We also used the mean average precision metric (mAP). This metric was specifically designed for the object detection task. This metric estimates the global performance of a model under multiple confidence thresholds. To calculate this metric, we pass by two stages. The first stage is the calculation of the Average Precision for every separate class. AP is defined by the the following equation:

[TABLE]

where

[TABLE]

with $p(\tilde{r})$ is the precision calculated at the recall $\tilde{r}$.

At the second stage, the mAP is concluded as the average of the separate APs calculated for every class. The formula is defined by the following equation:

[TABLE]

Where N is the number of classes. In real case, we don’t use the full range of confidence thresholds. For example, we use mAP(0.5) which is the mAP calculated for the confidence threshold of 0.5. Also, we use mAP(0.5:0.95) which is the mAP calculated for different confidence thresholds starting from 0.5 to 0.95, by a step of 0.05.

The figure 5 draws the progress of the metric mAP (0.5) during the training of YOLO v3 [53] on the constructed vehicle detection dataset.

In table 3, we put the performance metrics of the vehicle detection task in TAU. We put the statistics for both YOLO v3 and YOLO v3 tiny (which is a small version of the model designed for embedded and real time devices).

The metrics show a very high accuracy of the vehicle detection. This is mainly because the viewpoint from which we see the vehicles is the same for the whole dataset. This helps the model to grasp the pattern more accurately. Due to this high accuracy, all the insights based on it will have a similar accuracy. For example, the vehicle tracking task will be based on the bounding boxes estimated for the vehicle. There will not be identity switches because all the vehicles are moving on the same ground. Hence, we will have a good accuracy for the tracking task. Second, for vehicle assignment to zones, all the assignments will be based on the center of the bounding box of the vehicles. The accuracy of the zone assignment will be directly derived from the vehicle detection task. Thus, we will get a similar accuracy for it. This is similar for the other extracted insights.

4.2 Main insights extracted from the UAV video.

The principle of TAU was elementary: providing a robust and straightforward pipeline to automatically analyze traffic using UAV video. In the Figure 6 (the original video is presented here: https://youtu.be/kGv0gmtVEbI), we show an example of extracting two other insights from one UAV video. The figure represents four videos. The video on the top left corner represents the input UAV video taken on top of a crossroad. The video in the bottom left corner represents the results after applying the TAU framework to extract the seven main types of insights. The two plots on the right represent the extraction of two other types of insights: the histogram of vehicle velocity per frame and the histogram of vehicle size per frame.

To get better visualization on the output video generated by TAU, please see the following link: https://youtu.be/wXJV0H7LviU. For the remainder of the paper, this output video will be referenced as UAV test video. A screenshot of the video is shown in Figure 4.

It is clear that TAU generates the bounding box for every vehicle and estimates the velocity in km/h. Therefore, seven zones are defined by the user in this video. The number of vehicles inside every zone is updated after every frame.

4.3 Number of Vehicles that crossed every line apart from both directions.

If the Transport Engineer wants to study the vehicles’ crosses over specific lines, he can implement it based on TAU. It will give him three pieces of valuable information. The first is the number of vehicles’ crosses for the total UAV video. The second is the number of allowed crosses. The third is the forbidden crosses over this line. For example, Figure 7 shows that during 4:00 of the selected UAV video, a total of 29 crosses had been recorded: 27 are allowed, and two are forbidden.

4.4 The vehicle trajectories followed by all the vehicles during the recorded UAV video.

Based on TAU, we can draw all the trajectories followed by the vehicles inside a specific UAV video alongside their directions. For example, Figure 8 shows these trajectories for the UAV test video. It gives better insight into the vehicles’ movements.

4.5 The total number of vehicles in all the recorded area over time.

To know the level of traffic congestion, it is beneficial to know the number of circulating vehicles over time. For example, Figure 9 illustrates the curve of the number of vehicles by time for the UAV test video. The number of vehicles fluctuates between 112 and 176 in total.

4.6 The number of vehicles per zone over time.

Sometimes, it is advantageous to analyze the congestion level in specific zones and not in all the surveyed areas. This is to see if some zones are more crowded than others. For example, Figure 10 displays the curve of the number of vehicles for every zone by time for the UAV test video. We can see that zones 6 and 7 are the less crowded while zones 3, 4, and 5 are the most crowded. Zone 0 means the areas outside the zones of interest defined by the user (zones 1 to 7).

4.7 Average velocity of vehicles in all the recorded area over time.

It is handy for the Transport Engineer to analyze the average velocity in all the surveyed areas. This metric reflects the degree of congestion and the quality of the traffic. If it is low, so there are some problems to resolve. If it is high but below the maximum allowed velocity, the traffic quality is good. If it is above the maximum allowed velocity, many drivers are violating the rules. For example, Figure 11 illustrates the curve of the average velocity of all the vehicles circulating in the UAV test video over time. We can see that the average velocity fluctuates between 10 km/h and 78 km/h. This differs following the traffic lights indication for every road and the number of vehicles circulating in every zone. This detail will be emphasized more in the following sub-section.

4.8 Average velocity of vehicles per zone over time.

Sometimes, it is not fair to analyze the overall average velocity as the traffic conditions may change from one zone to another. Hence, it is better to analyze the average velocity for every zone apart. This helps to understand if some zones suffer from abnormal congestion that needs investigation. For example, we drew the average velocity per zone curve for the vehicles circulating in the UAV test video. We can see that zone 1 has the best traffic flow. We can see also that vehicles are circulating at zones 6 and 7 with low velocity. We note that sometimes the average velocity returns to zero due to traffic condition (traffic light is red, for example, which oblige all the vehicles to stop).

4.9 Number of Allowed and Forbidden crosses over time during the recorded session.

One of the most critical events in traffic is vehicles circulating in the wrong direction. This is dangerous driving behavior. Therefore, the Transport Engineer sometimes needs to go deeper in analyzing when this behavior occurs. This helps to understand the causes of such behavior and to resolve it. For example, Figure 13 shows the curves of the new allowed and forbidden crosses for every second for the UAV test video. We note, in general, that there is a correlation between the number of allowed and forbidden crosses. If the number of allowed crosses increases, the number of forbidden crosses also increases at the same pace. However, the total number of forbidden crosses increases unexpectedly between the seconds 140 and 160.

4.10 Correlation degree between the Average velocity and the number of vehicles over time.

Sometimes, the intuitive search for correlation between the different metrics is insufficient. Better is to use precise figures representing the different metrics and to use correlation metrics. For example, to interpret the relationship between the average velocity, the number of vehicles, and the vehicle sizes in the UAV test video, we used two methods. First, we drew their representation in the same figure (please see the Figure 14. Second, we calculated the Pearson correlation coefficient between them (Please see Table 4).

To remind the Pearson correlation coefficient, it is a real value that ranges between -1 and 1. The value 1 reflects a perfect positive correlation (like the correlation with the same factor). The value -1 reflects a perfect negative correlation. 0 means no correlation at all. Generally, the values from 0.50 to 1 mean a strong correlation. The values from 0.30 to 0.50 mean moderate correlation. The values between 0 and 0.30 mean low correlation. The meaning is also applicable in the negative scale for the degree of the negative correlation. From Table 4, we can see that there is a high correlation between the Number of Vehicles circulating and the Average velocity (Pearson correlation coefficient = 0.70). This means that the more the average velocity increases, the more the number of vehicles are circulating and the more the quality of traffic flow improves. This correlation is consolidated in Figure 14. On the other hand, there is a moderate correlation between these factors and the Vehicle sizes. Vehicle sizes do not interfere well with the quality of traffic flow in this case. Sometimes, some of these conclusions could be intuitive but proving them with measurable metrics is valuable.

4.11 Factors influencing the number of Allowed and Forbidden crosses over time.

To deeply analyze the factors leading to the increase of the forbidden crosses, the Transport Engineer can study the correlations between the new forbidden crosses on every second and the potential factors like the average velocity of vehicles, the average vehicle size, and the number of circulating vehicles. We applied this approach for the UAV test video and drew the different curves of these factors in Figure 15. The Pearson correlation coefficient between them is also calculated and displayed in Table 5.

Inside the Table 5, we will only consider the column of "New Forbidden Crosses" and analyzes one by one the correlation with every potential factor. No strong correlation can be seen with them. Only a moderate correlation is noted.

First, we have a moderate correlation with the Average Velocity (0.30), the number of vehicles (0.46), and the number of Allowed Crosses (0.34). An increase in one of these factors has a medium influence on increasing the Number of Forbidden Crosses. The table shows that there is a strong correlation between these three factors. Hence, having more vehicles on the roads increases the chance to get forbidden crosses but with a moderate impact.

Second, A moderate negative relation is noted with the Average Size of vehicles. This means that bigger vehicles tend to behave more reasonably than smaller ones. This is because the size of the vehicle has a moderate impact. For example, it reduces its tendency to go in the wrong direction of the roads.

4.12 Heat map of the maximum recorded velocity per pixel.

Sometimes, having a large dataset about the vehicle movements does not give a tangible service to the Transport Engineer. He cannot know how to interpret them to understand the traffic. Therefore, it is recommended to use heat-maps. They give a beautiful visualization of the data to understand the different metrics recorded deeply. Hence, this study used three different heat maps to visualize the extracted data better. This subsection draws the maximum recorded velocity per pixel heat-map for all the surveyed areas. It is applied on the UAV test video to get the Figure 16.

We can see in Figure 16 that the value 0 is recorded for all the zones outside the roads. The non-null velocities are recorded only on the road’s zone. It is also seen that the maximum recorded velocity varies from 20 km/h to 145 km/h. In some zones, the vehicles cannot surpass some velocity limits. Some vehicles reached very high velocities (more than 120 km/h). However, in general, the maximum velocity is an instant pic in the vehicle circulation and does not continue during all their trip. This is why we need to analyze the heat map of the average velocity to judge the vehicles’ acceleration behavior. This is done in the following subsection.

4.13 Heat map of the average velocity of vehicles per pixel.

The heat-map of the average velocity per pixel gives a precise measurement of the estimated velocity of one random vehicle circulating at every point on the roads. It is generated for the UAV test video to get the Figure 17. It is noted that in some zones, the vehicles’ movement is remarkably slower than in others. Vehicles tend to move very fast (above 60 km/h on average). The Transport Engineer could use this heat-map to ensure pedestrians’ safety. Some zones can have high average velocity, while we may have pedestrians crossing it innocently. He can reduce the average velocity and protect these pedestrians by installing speed bumps in the critical parts. Sometimes, many Transport Engineers does not have the right mean to decide where to put the speed bumps. TAU gives them fast and precise metrics to decide better where to put them using only one UAV video.

4.14 Heat map of the congestion level per pixel.

Another useful heat-map that the Transport Engineer can use is the heat-map of the congestion level per pixel on the road. It reflects how many vehicles passed on every specific pixel of the surveyed area. It is applied for the UAV test video to get the Figure 18. In most pixels, the value is 0, which means that no vehicles passed there. Also, most vehicles follow the same specific tracks on the roads. Most vehicles follow common tracks even for passing from one zone to another. This reflects that most vehicles respect the lines between tracks which is a good sign of driving awareness. Moreover, some specific tracks are very congested than others (more than 600 vehicles passed during the recorded video). Other tracks are rarely used (less than 50 vehicles passed during the recorded video). This gives clear insight into the congestion by focusing only on the congested tracks and trying to equilibrate the traffic load between all the tracks.

4.15 The diagram of vehicle movements between zones during the recorded UAV video.

Crossroad management remains one of the biggest headaches for the Transport Engineer. Intersecting roads represent the most critical part of every traffic area. Every road can be divided into many tracks. Every track could have its direction. traffic lights are normally used to allocate time for every possible move inside the crossroad. Allocating time for the possible moves over the roads is a complicated process. A minimal error in this allocation could generate a significant waste of time for many drivers. TAU comes with a simple and efficient method to resolve all these problems. The method is based on counting the percentage of all the vehicles movements inside the crossroads. It is better to estimate it in a normal traffic use. This method gives a self-explainable metric to decide the right rules to optimize the traffic. To give an example, the method is run on UAV test video to get the matrix represented in Table 6.

The Table 6 represents the percentage of the vehicle moves inside the crossroad represented in UAV test video. The row index represents the starting zone of the vehicle. Zones are indexed from Z1 until Z7. Z0 is not represented because it is the default zone outside the user-defined zones. It does not have any specific meaning for crossroad management. The column index is the destination zone of the considered vehicle. For example, from the table: 17% of the vehicles entering the crossroad pass from zone Z3 to zone Z2. The value is rounded to the nearest integer below the value. From this matrix, we drew a visually better diagram of the vehicles’ moves in $Figure$ 19.

In $Figure$ 19, the zones can be one of two types. The first type is the zones entering vehicles into the crossroad (Z1, Z3, Z4, and Z7). They are represented in blue color. Inside each of them, a blue arrow represents the percentage of vehicles entering from the selected zone into the crossroad. The second type of zones is the zones exiting the vehicles from the crossroad. They are represented in yellow color. Inside each of them, a yellow arrow represents the percentage of the vehicles exiting from the crossroad through the selected zone. The green arrows represent the percentage of the vehicle movements inside the crossroad. The thicknesses of drawn arrows follow the scales of the represented percentages. The diagram is drawn manually, but a software tool can be developed on top of TAU to generate it automatically from the UAV video like done for other types of insights (heatmaps, histograms, curves, and plots).

Using the Figure 19, the Transport Engineer can see in only one map all the traffic movements in the crossroad. Based on it, he can decide the different rules to set. For example, allocating time slots for the different traffic lights is not a headache like before. This allocation can be made following precisely the percentage of vehicle moves (Green Arrows).

TAU can be used for static crossroad management. Static means that the rules settings are not susceptible to continuous change. However, TAU is also useful for dynamic crossroad management. In dynamic crossroad management, the traffic rules can be changed continuously following the traffic conditions. For example, suppose the crossroad traffic is remarkably changed from one hour to another. In that case, the time allocation for the traffic lights can be changed following the new metrics recorded during that hour. A similar figure to Fig. 19 will be generated for every hour to the transport engineer to change the rules based on it automatically.

For both static and dynamic crossroad management, TAU has given excellent service to analyze and decide the best rules to choose in every condition. Moreover, TAU has given self-explainable metrics that alleviate the responsibility from the side of the Transport Engineer because it is totally based on the recording of vehicles’ moves on the roads.

4.16 The probability density function of vehicles sizes during the recorded UAV video.

It is valuable to know the distribution of vehicle sizes circulating in a surveyed area. This is why TAU has given the possibility to extract the Probability Density Function (pdf) of the vehicle sizes from the UAV video. This feature is tested on the UAV test video to get the Figure 20. Inside this figure, there are four extremums in the curve. The greatest extremum represents the most circulating vehicles in the video: sedan cars. The next three extremums are related to small-sized vehicles (like motorcycles), trucks, and buses. It is easy to generate the percentage of every size in the surveyed area from this pdf.

4.17 Probability density function of vehicles velocities during the UAV video.

TAU has also given the possibility to generate the pdf of vehicle velocities for one surveyed area. It gives the distribution of velocities recorded in the selected UAV video in one figure. This feature is tested on the UAV test video to get the Figure 21. We can see that the most recorded velocities are below 50 km/m. There were minimally recorded velocities between 50 km/m and 90 km/m. Sporadic cases are recorded for velocities more than 90 km/h. This feature is handy for the Transport Engineer to reasonably set the speed limit on the roads without affecting the traffic’s fluidity too much.

4.18 The histogram of vehicle sizes distribution per frame

Sometimes, getting one pdf for the total video hides important changes occurring during the time. This is why TAU has given the possibility to generate histograms for vehicle sizes distribution during the time. We found that representing the vehicle size distribution using a histogram is more appropriate than pdf to track changes over time easily. An example of this feature applied on the UAV test video is presented in $Figure$ 6 (video: https://youtu.be/kGv0gmtVEbI). The histogram of the vehicle sizes is displayed in the lower right corner of the Figure. We can see that the global pattern of the histogram follows, in most cases, the full pdf represented earlier. The same distribution of the vehicle size is kept, although there are continuous fluctuations. This can guide the Transport Engineer in setting the different sizes of vehicles using the traffic and the good rules to optimize it.

4.19 The histogram of vehicle velocities distribution per frame

TAU generates histograms for the distribution of vehicle velocities over time. An example of this feature applied on the UAV test video is presented in $Figure$ 6 (video: https://youtu.be/kGv0gmtVEbI). The histogram of the vehicle velocities is displayed on the right top corner of the Figure. This feature is handy for the Transport Engineer to study the different velocities recorded over time in the surveyed area. It helps estimate the number of vehicles breaking the speed limit and the number of vehicles respecting it. It can give a clear insight into the utility of setting new speed limits or removing old ones.

4.20 Inference speed and hardware architecture

TAU is tested on a sample 4K video (resolution is 3840 × 2160). This is to measure the inference speed. The chosen sample video is taken from a fixed UAV at 175 meters from the ground. The camera view is completely perpendicular to the ground. The input image is divided into 40 images of size $512*512$ during the processing. Every image passes into the YOLO v3 network to catch the bounding boxes around every detected vehicle on the ground. Then we conclude the global parameters to be passed into the DeepSORT algorithm. During the generation of the output video, the inference speed of the YOLO v3 is always fixed. However, the inference speed of DeepSORT fluctuates following the number of vehicles. The more vehicles we have, the more we need processing power to calculate the Kalman predictions and the appearance descriptor for every vehicle. However, the overhead of the DeepSORT is far lesser than the one of YOLO v3. The machine that we used during the inference has the following specifications:

CPU: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz

• 2.

GPU: NVIDIA GeForce GTX 1060 6 GB VRAM

• 3.

Memory: 32 GB

• 4.

CUDA version: 9.1

• 5.

Framework: Tensorflow 1.14 (with Keras API)

We found that the average inference time needed to predict one frame is 4.3 seconds, corresponding to 0.23 FPS (Frames per Second). As the input video has 24 frames per second, processing one second of the UAV video costs one minute and 43 seconds. Therefore, we need 103 hours to process one hour from the UAV video. This high processing cost is due primarily to the YOLO v3 bottleneck. This bottleneck can be solved by working on larger input sizes, parallelizing the processing, or considering fewer frames to process per second.

4.21 Limitations in the current version of TAU (TAU v1).

During the test of the different features of TAU, we noted that there are some limitations to be targeted in the next version of the Framework (TAU v2):

Algorithms 1 & 2 are not optimal: Although Algorithms 1 & 2 did the job well, one drawback needs investigation. This drawback is the discontinuity of the metrics in pixels of coordinate $x$ or $y$ that are multiples of the size considered in processing. In the test, it was 512. This is the reason for the discontinuity in the metrics where vehicles pass by pixels multiple of 512. Both Algorithms 1 & 2 need to be modified to overcome this flaw.

• 2.

TAU cannot run online in high resolutions: In the current version, TAU can be run online only for resolutions that are less than $608*608$. The processing becomes a lot heavier with greater resolutions. Therefore, parallelizations method are needed, especially for the heavier part of the processing (Algorithm 2). We need to divide the processing over multi-GPUs to get the different Vehicles Bounding Boxes for every processed frame. Then the Bounding boxes should be recalculated before running the Kalman filter and generating the appearance descriptors for every detected vehicle. Finally, the whole pipeline should be optimized to run on a multi-GPUs system to ensure the TAU framework’s real-time running.

• 3.

TAU should take consideration for inclined vehicles: If the vehicle movement is following the $\vec{x}$ axis or $\vec{y}$, the bounding box is fitting the vehicle area allowing to adopt many approximations made in this study. However, if the vehicle moves following an inclined axis, the bounding box will not fit the vehicle, making the vehicle size calculation inappropriate. However, the velocity estimation remains efficient even in these conditions.

5 Conclusion

This study introduced a robust and straightforward framework for traffic Analysis from UAV video (TAU). First, the scientific context of the study has been introduced. Then, the building blocks of the TAU have been described. After that, the theoretical basis of the seven main insights extracted using the TAU from the UAV video have been detailed. It has been proven that the built Framework can be easily enriched to extract more insights. In this study, 17 other insights are taken as an example. In the experimental part, a total of 24 insights from an UAV video sample have been discussed. We detailed the utility of these features for a transport engineer in his mission to optimize traffic fluidity, reduce waste of time and ensure safety on the roads. We tried to emphasize the main goal of TAU, which is to give the easiest way for a transport engineer to efficiently analyze and manage traffic by using only one UAV video, no more.

However, TAU has some limitations in its current version. The main limitation is the discontinuity of the metrics on the x-axes where the pixel index is multiple of the width of the processed frame, and also on the y-axes where the pixel index is multiple of the height of the processed frame. The second main limitation is the inability to run online for high resolutions.

Beyond solving these limitations, TAU can be enriched by the addition of new features. For example, we can incorporate the predicted trajectory for every vehicle and estimate the recommended velocity for every vehicle in real-time. Also, architectural improvements can be made in TAU. For example, the chosen vehicle detector YOLO v3 [53] can be substituted by a more recent generic object detector like YOLO v4 [14] or a designed object detector for aerial images specifically like the Butterfly detector [1]. Also, the vehicle tracker Deep SORT can be replaced with a more recent generic multi-object tracker (MOT) [22].

To extend TAU, we also remind the recommendation made in the introduction (Section 1): combining the use of UAVs and stationary cameras are the best approach to benefit from the advantages of both tools and overcome their limitations. For example, synchronizing TAU with a vehicle plate identification station installed on the road helps assign the real identity to every tracked vehicle. This opens the doors toward new possibilities like gathering information about the vehicle id, age, brand, and history. Then, better analysis and management of the traffic can be implemented.

In the next version, TAU can also be extended by adding predictive analysis. An AI model can be developed to learn how to predict approximately the traffic status by learning from daily traffic data. This prediction greatly helps the traffic managers to avoid congestion and other traffic anomalies before happening. This helps more to discover the main factors that impact traffic fluidity for a better understanding of traffic.

Apart from predictive analysis, TAU can be extended by enriching its ability to work on camera views. It is useful to investigate more the intrinsic and extrinsic parameters of the camera mounted in the UAV. This is to be able to deduce a strong foundation to extract traffic parameters from a camera view that is not perpendicular to the ground. Moreover, TAU should be assessed to work on the different weather conditions (day/night, sunny/rainy, etc.). This is primordial to ensure normal daily use.

Acknowledgments

The authors would like to acknowledge the support of Prince Sultan University for funding this study. Special acknowledgement to Robotics and Internet of Things Lab at Prince Sultan University, Riyadh, Saudi Arabia.