Machine Learning-Based Delay-Aware UAV Detection and Operation Mode Identification over Encrypted Wi-Fi Traffic

TL;DR

This paper presents a machine learning framework that detects and identifies consumer UAVs and their operation modes over encrypted Wi-Fi traffic, addressing limitations of physical detection methods in civilian environments.

Contribution

It introduces a novel feature extraction and selection method using packet size and inter-arrival time, optimized for fast detection and mode identification over encrypted traffic.

Findings

High detection accuracy achieved in real-world Wi-Fi traffic

Effective identification of UAV operation modes

Reduced online detection delay

Abstract

The consumer UAV (unmanned aerial vehicle) market has grown significantly over the past few years. Despite its huge potential in spurring economic growth by supporting various applications, the increase of consumer UAVs poses potential risks to public security and personal privacy. To minimize the risks, efficiently detecting and identifying invading UAVs is in urgent need for both invasion detection and forensics purposes. Given the fact that consumer UAVs are usually used in a civilian environment, existing physical detection methods (such as radar, vision, and sound) may become ineffective in many scenarios. Aiming to complement the existing physical detection mechanisms, we propose a machine learning-based framework for fast UAV identification over encrypted Wi-Fi traffic. It is motivated by the observation that many consumer UAVs use Wi-Fi links for control and video streaming. The proposed framework extracts features derived only from packet size and inter-arrival time of encrypted Wi-Fi traffic, and can efficiently detect UAVs and identify their operation modes. In order to reduce the online identification time, our framework adopts a re-weighted $\ell_1$-norm regularization, which considers the number of samples and computation cost of different features. This framework jointly optimizes feature selection and prediction performance in a unified objective function. To tackle the packet inter-arrival time uncertainty when optimizing the trade-off between the detection accuracy and delay, we utilize Maximum Likelihood Estimation (MLE) method to estimate the packet inter-arrival time. We collect a large number of real-world Wi-Fi data traffic of eight types of consumer UAVs and conduct extensive evaluation on the performance of our proposed method.

Tables

Table 1. Table I: Statistical features (sample size N = 100 𝑁 100 N=100 ).

Function: Feature Name	Description	Comput. time
$V_1\left(x\right)$: mean	$\overline{x}=\frac{1}{N}{\sum }_{i=1}^{N}x\left(i\right)$	0.672 $\mu s$
$V_2\left(x\right)$: median	The higher half value of a data sample.	4.365 $\mu s$
$V_3\left(x\right)$: MedAD11footnotemark: 1	$MedAD=median\left(\left|x\left(i\right)-median\left(x\right)\right|\right)$	8.346 $\mu s$
$V_4\left(x\right)$: STD22footnotemark: 2	$\sigma =\sqrt{\frac{1}{N-1}{\sum }_{i=1}^N{\left(x\left(i\right)-mean\left(x\right)\right)}^2}$	1.608 $\mu s$
$V_5\left(x\right)$: Skewness	$\gamma =\frac{1}{N}{\sum }_{i=1}^N(x\left(i\right)-mean\left(x\right))/\sigma ){}^3$	14.917 $\mu s$
$V_6\left(x\right)$: Kurtosis	$\beta =\frac{1}{N}{\sum }_{i=1}^N{\left(x\left(i\right)-mean\left(x\right)/\sigma \right)}^4$	14.095 $\mu s$
$V_7\left(x\right)$: MAX	$H=\left({Max\left(x\left(i\right)\right)|}_{i=1\dots N}\right)$	0.464 $\mu s$
$V_8\left(x\right)$: MIN	$L=\left({Min\left(x\left(i\right)\right)|}_{i=1\dots N}\right)$	0.652 $\mu s$
$V_9\left(x\right)$: Mean Square	$MS=\frac{1}{N}{\sum }_{i=1}^N{\left(x\left(i\right)\right)}^2$	1.147 $\mu s$
$V_{10}\left(x\right)$: RMS	$RMS=\sqrt{ms\left(x\right)}$	1.273 $\mu s$
$V_{11}\left(x\right)$: PS33footnotemark: 3	$3\left(mean\left(x\right)-median\left(x\right)\right)/\sigma$	8.011 $\mu s$
$V_{12}\left(x\right)$: MAD44footnotemark: 4	$MAD=\frac{1}{N}{\sum }_{i=1}^N\left|\left(x\left(i\right)-mean\left(x\right)\right)\right|$	2.531 $\mu s$

Table 2. Table II: Empirical and exponential cumulative distribution function (CDF) goodness-of-fit statistics.

	UAV type
	Bebop 1	Bebop 2	Spark	UDI
KS11footnotemark: 1	0.0612	0.0508	0.0773	0.0811
CvM22footnotemark: 2	0.0720	0.0633	0.0691	0.0794
	UAV type
	Discovery	Tello	TDR	Wingstand
KS	0.0801	0.0622	0.0910	0.06741
CvM	0.0865	0.0890	0.0533	0.0695

Table 3. Table III: Tested UAVs’ identification performance.

Traffic	$E\left[p^{\ast }\right]$	$E\left[t_{p^{\ast }}\right]\left(ms\right)$	Accuracy (%)
Bebop 1	87 ($\pm$8)	160.43 ($\pm$10.01)	87.84 ($\pm$1.20)
Bebop 2	95 ($\pm$13)	151.91 ($\pm$18.82)	90.75 ($\pm$1.74)
Spark	93 ($\pm$11)	142.80 ($\pm$15.57)	95.23 ($\pm$0.69)
UDI	141 ($\pm$21)	350.79 ($\pm$23.41)	85.76 ($\pm$2.38)
Discovery	94 ($\pm$3)	131.11 ($\pm$10.42)	92.52 ($\pm$0.85)
Tello	72 ($\pm$7)	121.65 ($\pm$31.30)	93.68 ($\pm$1.01)
TDR	68 ($\pm$13)	100.77 ($\pm$12.75)	89.66 ($\pm$2.10)
Wingsland	75 ($\pm$18)	92.46 ($\pm$ 21.22)	94.39 ($\pm$1.85)

Equations

$$\displaystyle J\big{(}\mathbb{\tilde{x}}(t_{k})\big{)}=C_{1}\big{(}h_{\gamma}^{k}(\mathbb{\tilde{x}}(t_{k}),\tilde{y})\big{)}+C_{2}(t_{k}).$$

$$\displaystyle p^{*}=\underset{p\in\left\{k,...,n\right\}}{\arg\min}\quad J\big{(}\mathbb{\tilde{x}}(t_{p})\big{)},$$

$$\min_{W^{j}}\;\mathcal{L}\big{(}y,h_{\gamma}^{j}(\mathbb{X}^{j},W^{j})\big{)}+\sum_{i=1}^{2l}\lambda_{i}^{j}|W_{i}^{j}|,$$

$$E^j(X_i^j)=\sum _{y_i\in \text{pazocal}Y}P(y_i|X_i^j)$$

$$\displaystyle C_{1}\big{(}h_{\gamma}^{k}(\mathbb{\tilde{x}}(t_{k}),\tilde{y})\big{)}=\sum_{i=1}^{m}f_{w_{i}}^{k}E^{k}(X_{i}^{k}).$$

$$\displaystyle\mathcal{L}(\mu_{i};\tau_{1}^{i},...,\tau_{n}^{i})=\prod_{j=1}^{n}f_{\pazocal{T}^{i}}(\tau_{j}^{i};\mu_{i})=\mu_{i}^{n}\text{exp}\Bigg{(}-\mu_{i}\sum_{j=1}^{n}\tau_{j}^{i}\Bigg{)}.$$

$$\text{boldmath}l({\mu }_i;{\tau }_1^i,{\tau }_2^i,...,{\tau }_n^i)=n\text{ln}({\mu }_i)-{\mu }_i\sum _{j=1}^n{\tau }_j^i.$$

$${\hat{\mu }}_i=\text{arg}\underset{{\mu }_i}{\max }\text{boldmath}l({\mu }_i;{\tau }_1^i,{\tau }_2^i,...,{\tau }_n^i),$$

$${\hat{\mu }}_i=\frac{n}{{\sum }_{j=1}^n{\tau }_j^i}\text{for}i=1,...,\upsilon .$$

$$C_2(t_p)=\sum _{i=k}^p{\tilde{\tau }}_{i+1}\text{for}p=k,...,n$$

$$MS{E}_p=\frac{1}{n-p}\sum _{i=p}^n({\tau }_{i+1}-{\tilde{\tau }}_{i+1}{)}^2\text{for}p=k,...,n$$

Full text

Index Terms -** Unmanned aerial vehicle (UAV) detection, machine learning, encrypted Wi-Fi traffic classification.**

I Introduction

In the past few years, we have seen a significant growth of the consumer unmanned aerial vehicle (UAV) market for personal recreation. Despite its huge potential in spurring economic growth, the significant increase of consumer UAVs raises lots of issues regarding airspace management, public security, and personal privacy [1]. It was reported that an Army chopper was struck by an illegally flying drone over a residential neighborhood in September 2017 [2]. In April 2016, a UAV was peeping outside a teenager’s bedroom window in Massachusetts [3]. In January 2015, a small UAV crashed on the White House lawn bringing the worry about security measures [4].

To deal with these threats, consumer UAV registration mechanisms, started by Federal Aviation Administration (FAA), have been promoted worldwide, which can help law enforcement officials to handle the UAV and its owner information [5]. UAV-restricted zone and geo-fencing are requested to set up in sensitive areas, such as airports, nuclear facilities, and data centers, to protect them from hostile UAV invasion.

However, the enforcement of regulations is not an easy task in practice. Plenty of UAVs are still unregistered, and many UAVs do not have geo-fencing or the geo-fencing can be turned off easily. There is an urgent need to quickly detect an intruder UAV in a restricted area, or assist the forensics investigation to identify its appearance and operation mode. An ideal detection technique should give us the alert when the restricted area is invaded by unwanted UAVs at the earliest stage. After that, counter-measures for intruder UAVs can be applied and the UAV owner may be tracked or located. Therefore, how to efficiently detect the consumer UAVs is of utmost importance.

Other than detecting UAVs, identifying UAVs’ operation mode will be very useful for forensics purposes. Being able to identify the operation mode of intruder UAVs can help investigators to restore the course of the incidents, which could be used as court evidences in a legal process and help law enforcement officials to improve countermeasures or responses to various possible UAV incidents.

Many physical detection mechanisms, such as radar [6, 7], acoustic [8, 9], and vision [10, 11, 12], have been proposed for UAV detection. When using only one of these sensors for detection, these methods may get less effective in some practical scenarios, especially in a crowded urban environment. The radar signals may get blocked by walls, buildings, and other obstacles, which are very common in a civilian environment. The vision detection technique cannot detect the UAV in non-line-of-sight scenarios and dark. The acoustic detection can be interfered by the environment noises which may overwhelm the relative small sound produced by tiny rotor-craft or gliding fixed-wing UAV.

Aiming to complement the above conventional physical detection mechanisms, we propose to explore machine learning-based Wi-Fi traffic identification approaches to achieve fast UAV detection and operation mode identification. It is motivated by the observation that many existing consumer UAVs are equipped with Wi-Fi interfaces and communicate with a user handheld device (e.g., smartphone) for command control or video streaming. Detecting UAVs through wireless traffic identification brings us several advantages over existing mechanisms. First, Wi-Fi signal sensing and packet capturing are less affected by obstacles, other flying objects, acoustic noise, or light conditions that could affect physical detection mechanisms. Second, Wi-Fi data traffic provides cyber information about UAVs’ type and their operation mode, which can be very useful for forensics investigation.

Challenges: At the same time, UAV detection through Wi-Fi traffic identification introduces unique challenges that separate it from traditional traffic identification [13, 14, 15] and sensing tasks as follows:

1. UAV traffic can be encrypted. Therefore, existing network monitoring and intrusion detection mechanisms that are based on packet header examination or port filtering are not applicable to encrypted UAV traffic. For example, Wi-Fi controlled UAVs (such as DJI and Bebop drones) use WPA2 to secure the wireless communication. Although SSID in the MAC frame may reveal information about the type or vendor of the drone, it can be easily changed through drone control apps.
2. Existing machine learning methods cannot be directly used to identify UAV traffic in a timely manner. For real-time applications, we need to identify the UAV as soon as it is appearing in or approaching to a restricted area. From learning and classification perspective, traditional machine learning methods [13, 14] that only aim at minimizing detection error cannot be directly applied. Detection delay introduced by the computations on feature generation and future packet arrival time should also be considered.
3. Traditional time series early detection strategies [16] cannot be applied to UAV traffic. The inter-packet arrival time of UAV traffic is random, so the traditional time series early detection method which is based on fixed time intervals cannot be directly applied.

To address the above challenges, we propose a delay-aware machine learning-based UAV detection framework to strike a tunable balance between UAV detection accuracy and delay. Our classification framework treats the encrypted data flow as a time series and extracts statistical features only based on the packet size and inter-arrival time. By considering the computation time among different features, our framework adopts a re-weighted $\ell_{1}$-norm regularization and integrates feature selection and performance optimization in one objective function. To tackle the packet inter-arrival time uncertainty when estimating the delay cost function, we use maximum likelihood estimation (MLE) method to estimate the packet inter-arrival time. Finally, expected total cost function integrates misclassification/misdetection and delay cost which are updated online when a new packet arrives and an optimal detection decision is made to minimize the expected total cost function.

Our main contributions are summarized as follows:

• •

We propose a machine learning-based framework to achieve delay-aware UAV detection and operation mode identification over encrypted Wi-Fi traffic. This framework extracts features derived only from information of packet size and inter-arrival time. This framework can be applied to other types of encrypted traffic, such as cellular traffic or proprietary protocol traffic as long as the packet size and interval can be measured.

• •

In order to reduce the model prediction time for fast UAV detection, our framework adopts $\ell_{1}$-norm regularization and integrates feature selection and accuracy optimization in one objective function, which considers the feature importance and difference of computation time among different features.

• •

We propose to use model-based MLE method to estimate the packet inter-arrival time. Then using the mean square error (MSE) as a well-known metric, we evaluate the performance of the estimation on the collected real-world dataset.

• •

Other than detecting and identifying different types of UAVs, our proposed method further identifies the UAV’s operation mode, such as standby, hovering, flying, etc.

• •

We collect a large amount of real-world encrypted Wi-Fi data traffic of non-UAV and eight types of consumer UAVs, and conduct extensive evaluations on the performance of the proposed methods.

Through comprehensive study, we obtain the following findings:

• •

The UAV traffic presents different patterns from non-UAV traffic. Therefore, machine learning based methods work well to differentiate UAV traffic from a wide range of non-UAV traffic.

• •

Due to vendor specific implementation of UAV command control and video streaming protocols, different types of UAVs present different traffic patterns which can be used to classify UAVs from different vendors.

• •

The UAV Wi-Fi traffic presents different patterns under different UAV operation modes. This finding implies a strong correlation or coupling between cyber information (data traffic) and physical information (operation mode) of UAVs. This finding is expected to motivate new cyber-physical defense and forensics mechanisms that leverage this cyber-physical coupling. We believe this methodology can be applied to other cyber-physical systems (CPS) and motivate more in-depth study on cyber-physical attack co-detection or co-defense for many Internet of Things (IoT) applications, such as connected cars, smart home, smart healthcare, and industrial control systems.

The rest of the paper is organized as follows. Section II discusses the related works. Problem setup is described in Section III. Section IV presents our proposed method on delay-aware UAV early detection and operation mode identification. Collection and analysis of real-world dataset is described in Section V. Extensive performance evaluation is conducted in Section VI. Section VII provides more detailed discussion on practical aspects of the proposed methodology and possible future work in this area. Section VIII concludes the paper.

II Related Work

II-A UAV Detection Mechanisms

In this section, we briefly describe the existing UAV detection methods.

II-A1 UAV Detection Through Physical Sensing

Existing UAV detection mechanisms mainly focus on physical sensing through various means, including radar, vision, and acoustic.

Radar system is one of the well-known and oldest techniques in aircraft detection dating back to World War II. In order to adapt the detection of small size UAVs, X-band radar systems were proposed [17, 18]. However, in the metropolitan areas (e.g., a city) radar based detection may become less effective due to its line-of-sight requirement [19]. The vision-based UAV detection based on video cameras [20] has the same weakness as radar based techniques, as it also requires line-of-sight between the camera and UAV. However, if the cost is not a much concern, building a detection system based on multiple radars and cameras fusion to cover the targeted area would be a reliable and promising UAV detection system.

The acoustic signal-based UAV detection is a method that can solve the out-of-sight problem [8, 9, 21]. However, this method has its own drawbacks as well. First, the acoustic signal coming from the UAV can be quite noisy due to the noise generated at the motors of electric-powered rotor-craft with fixed wings [22]. Second, other similar acoustic signal generating devices such as electric weed whackers can generate sound signals quite similar to UAV’s. In order to overcome the drawbacks of individual techniques, hybrid solutions have been proposed by combining the acoustic sensor and video camera [23]. Another hybrid solution incorporates the radar sensor as well [24].

II-A2 RF-fingerprinting Based UAV Detection

Recently, Zhao et al. [25] proposed a new method of RF signal fingerprinting in order to detect and identify the type of the UAVs. To do that, they propose to use Auxiliary Classifier Wasserstein Generative Adversarial Networks (AC-WGANs) based on the wireless signals collected from various types of UAVs. According to their results, this method can detect the UAVs in indoor and outdoor environments with average accuracy of 95% and 80%, respectively. In the other recent work [25], Bisio et al. [26] proposed a Wi-Fi statistical fingerprint-based amateur UAV detection method by applying existing multiclass classification machine learning algorithms. In this work the detection delay is not a concern, and thus, the main goal is to train a machine learning model to detect the intruding UAV based on the predefined and fixed number of statistical features which are computed in every fixed window size. Our proposed method considers detection delay and strike a tunable balance between detection accuracy and delay as well as feature computation time. Ezuma et al. [27] proposed a new detection and classification of micro-UAVs using RF fingerprints of the signals transmitted from the controller to the micro-UAV. In their technique, they utilized wavelet domain analysis to remove the bias in the signals which also helped in the processing data size reduction. For the classification purposes, a naive Bayes approach has been applied to distinguish the UAV signal frames from the non-UAV classes. In the testing phase, a signal energy level detection is also integrated to improve the detection performance. In average, the micro-UAV detection accuracy of 96.3% is achieved under various signal-to-noise ratio (SNR) levels on the channel. We believe integrating our work with the RF fingerprinting method proposed by Ezuma et al. [27] would result in a delay-aware, more robust, and accurate UAV detection system as each method could be a very suitable complement to the other.

A recent work [28] has been proposed to detect the approaching of a UAV within a short distance through the observation of received signal strength (RSS) changes of Wi-Fi signals. However, an intruder UAV may just launch inside a restricted area and hover, or standby on a neighboring roof to spy on someone. In these scenarios, this method will not work. Moreover, a changing RSS is not necessarily introduced by UAVs, but could be other moving objects with Wi-Fi interfaces, such as mobile users carrying smartphones or a driving car equipped with Wi-Fi connections. So the application scenario of this proposed technique is limited.

In another work [29], the authors propose a new RF-based drone detection method based on the physical characteristics of the drone, such as body vibration and body shifting, which impact the wireless signal transmitted by the drone during the communication. This method is not useful when the UAV is in the standby mode. Moreover, both [28] and [29] require line-of-sight connection between the RF signal monitoring system and UAV. Our proposed method based on Wi-Fi traffic identification relaxes this strong assumption.

Recently, Sciancalepore et al. [30] proposed to detect drones’ status, flying or laying on the ground, using encrypted network traffic identification. Three standard binary classification algorithms, Trees-J48 (J48), random forest (RF), and neural networks (NN) are applied to a 3DR SOLO UAV traffic dataset to identify the drone status. Different from this work, in our work we identify eight different operation modes as well as we propose a machine learning framework which addresses the delay-aware UAV detection problem. Our detection system is also tested in the presence of non-UAV data traffic to detect the UAVs used in the experiment.

II-B Data Traffic Classification/Identification

Classical approaches such as port-based, payload-based and deep packet inspection can be used to identify the type of the non-encrypted network data traffic. However, nowadays many application data traffic are encrypted for security purposes, and our work is closely related to encrypted data traffic classification/identification. There are several works for identifying the encrypted data flow based on protocol data fingerprinting in wired and wireless networks, where commonly a combination of statistical and machine leaning approaches have been used [13, 31, 14, 15, 32].

In [13], a new support vector machine (SVM) based method is proposed to identify three types of traffic, HTTP, File Transfer Protocol (FTP), and Email. One of the pioneering works in this area applies classification techniques to classify traffic in a wired network into classes of bulk transfer, small transactions, and multiple transactions [14]. Bernaille et al. [15] show that it is possible to distinguish the behavior of an application from the observation of the size and the direction of the first few packets of the Transmission Control Protocol (TCP) connection. In this work, three classical clustering algorithms, K-Means, Gaussian Mixture Model and spectral clustering are applied on the dataset to identify the flow. However, this method requires packet header traces analysis, and initial TCP connection packets. Xie et al. [31] proposes a new method called subspace clustering technique (SubFlow), which learns the intrinsic statistical features of each application to classify and identify the flow.

However, our work is different from the existing traffic identification works in the following aspects:

1. Our model provides packet-by-packet analysis, hence the decision is made in a timely manner as packets enter the detection system.
2. Our model adaptively finds the optimal number of the packets that are needed for optimal identification with high accuracy, while considering time cost (or delay).
3. When training the model, feature generation time is also considered and critical features are selected. Therefore, in the prediction/detection, useless features are not generated, and thus the detection delay is reduced.

Compared to our earlier work [33], in this paper, operation mode of the detected UAV is identified. To do so, we collect a large amount of real-world operation mode data traffic of four UAVs and apply multiclass classification machine learning algorithms to identify the modes. We also extend our delay-aware UAV detection test from four to eight commonly used consumer UAVs and conduct extensive evaluations on the performance of the proposed methods. Moreover, we provide performance evaluation for the packet inter-arrival time estimation using MLE. The results indicate that mean square error (MSE) of estimation is reduced when the information of a large number of packets are available in the detection system.

Note that our detection system is applicable to detect the intruding UAVs controlled by a user handheld device (e.g., smartphone). In other words, communication link between the UAV and controller should be established in order to monitor the traffic and detect the UAV by the proposed method. On the other hand, our method will be ineffective if the intruding UAV is equipped with the advanced autonomous systems such as Autonomous Guidance, Navigation and Control (GN&C) where no ground control station is required for command and control.

III Problem Setup

III-A System Setup

There is a Wi-Fi signal sensing and packet capturing system that can collect all the Wi-Fi traffic within a physical sensing range in real-time. There can be multiple Wi-Fi users in the sensing range and multiple UAVs or non-UAV devices. The sensing system may capture non-encrypted packets, for which we assume the system can tell the application types of the corresponding flow by examining the packet headers or contents. Since these non-encrypted packets can be easily identified by existing methods, we will only focus on encrypted Wi-Fi traffic in this paper. For an encrypted Wi-Fi frame, the information we can obtain about the frame is its source and destination MAC addresses, transmitter and receiver MAC addresses, packet size, and packet arrival time together with other MAC header information, such as frame type (control, management, or data), sequence number, and duration/connection ID.

In this paper, we mainly focus on two sequential tasks.

1. Wi-Fi controlled UAV detection: There are a large body of such kind of UAVs on the market, such as DJI, Bobop, DBPower drones, and etc. We assume the Wi-Fi communication between the drone and controller (e.g., smartphone) is encrypted using security protocols, such as WPA2. We assume a drone restricted area (i.e., No Drone Zone) which no UAVs are allowed to enter and operate (see Fig. 1). Our detection system can be implemented in the center of UAV restricted zone to monitor the area and detect any approaching UAVs as quickly as possible with a high accuracy.
2. UAV operation mode identification: For any detected UAV types in the first step, further identify its operation mode. Operation mode consists of standby, hover, forward, backward, and etc. It is noted that, if in case, a malicious spying UAV just get turned on or launched inside the restricted zone and stay on in the standby or hovering mode to accomplish the spying mission, our detection system not only can detect the presence of the spying UAV, but it also can identify in which mode the spying UAV is operating.

For the UAV detection and operation mode identification, we only use data frames. We divide the encrypted Wi-Fi traffic into individual flows according to the pair of source and destination MAC addresses. A unique flow includes the packets between a pair of nodes. The traffic in a flow can be bi-directional or unidirectional. In a real-time scenario, these flows usually interleave with each other in time. The goal of this paper is to identify the UAV data flows when frames are captured and decide the UAV type and its operation mode in a quick manner with high accuracy.

III-B Delay-aware UAV Detection Problem Formulation

The UAV detection over encrypted Wi-Fi traffic can be formulated as a machine learning classification problem. Let’s assume that we can obtain a large training dataset with $m$ flow traces with each trace having $n$ consecutive packets. The traces contain UAV and non-UAV flows which are labeled with their corresponding flow types $y_{i}\in\pazocal{Y}$, where $\pazocal{Y}=\left\{\text{UAV}_{1},\text{UAV}_{2},...,\text{UAV}_{\upsilon-1},\text{non-UAV}\right\}$ and $\upsilon=|\pazocal{Y}|$ denotes the number of class types in set $\pazocal{Y}$. $\text{UAV}_{j}$ for $j\in\left\{1,...,\upsilon-1\right\}$ denotes the UAV type $j$.

Packet size and packet inter-arrival time are two key attributes we extract from these traces for UAV detection and operation mode identification. The sequences of packet size and packet inter-arrival time for the $i$th trace are denoted by $\mathbf{x}_{i}$ and $\boldsymbol{\tau}_{i}$, respectively. Now, let $\mathbf{x}_{i}=(x_{i,1},\cdots,x_{i,n})$, where $x_{i,j}$ for $i=1,...,m$ and $j=1,...,n$ indicates the size of the $j$th packet in the $i$th trace. Similarly, let $\boldsymbol{\tau}_{i}=(\tau_{i,1},\cdots,\tau_{i,n})$, where $\tau_{i,j}$ denotes the inter-arrival time between the $j$th and $(j+1)$th packets ($j<n$) in the $i$th trace. Define a finite set $S=\left\{((\mathbf{x}_{i},\boldsymbol{\tau}_{i}),y_{i})\right\}_{i\in\left\{1,...,m\right\}}$ where the pair $(\mathbf{x}_{i},\boldsymbol{\tau}_{i})$ represents the packet size and inter-arrival time of the $i$th trace in set $S$, respectively.

Let $\mathbf{\tilde{x}}(t_{k})$ denotes the received incoming traffic up to its $k$th packet arrived at time $t_{k}$. Assume a set of multiclass classifiers $\mathcal{H}=\left\{h^{j}_{\gamma}\right\}_{j\in\left\{1,...,n\right\}}$ are trained to classify the incoming traffic flow $\mathbf{\tilde{x}}(t_{k})$, where $\gamma\in\pazocal{Y}$. When the Wi-Fi sensing system receives a new packet of the incoming traffic flow $\mathbf{\tilde{x}}(t_{k})$, its new features are extracted and incorporated in the prediction system. Intuitively, as more packets arrive, more accurate information about the traffic can be gained. On the other hand, collecting more packets introduces longer identification delay. Therefore, there is a trade-off between detection accuracy and delay in the detection process.

Let $C_{1}(\hat{y},\tilde{y}):\pazocal{Y}\times\pazocal{Y}\rightarrow\mathbb{R}$ denotes the test misclassification cost function where ($\hat{y}$, $\tilde{y})\in\pazocal{Y}$, and $\hat{y}=h_{\gamma}^{j}(\mathbb{\tilde{x}}(t_{j}))$ is the predicted class label, while the true class label of the incoming flow is $\tilde{y}$. Let $C_{2}(t_{p})\in\mathbb{R}$, $p>k$, be the time cost function which indicates the time cost value if UAV detection is postponed up to time instant $t_{p}$. Thus, the estimated total cost function is given by

[TABLE]

In order to find an optimal trade-off between the detection accuracy and delay, the total cost function $J$ needs to be minimized. Hence, we formulate the delay-aware UAV detection optimization problem as follows:

[TABLE]

where $p^{*}$ indicates the optimal number of the packets that needs to be received from the incoming flow before performing the UAV detection decision, and $t_{p^{*}}$ denotes the $p^{*}$th packet arrival time. When $p^{*}=k$ is the solution of the optimization problem in (2), there is no need to collect more packets because J\big{(}\mathbb{\tilde{x}}(t_{k})\big{)}<J\big{(}\mathbb{\tilde{x}}(t_{q})\big{)} for $q=k+1,...,n$. Therefore, the detection is performed instantly at time $t_{k}=t_{p^{*}}$. However, when $p^{*}\neq k$ is the solution of (2), it means that the total cost is minimized for $k+1\leq p^{*}\leq n$, thus the detection process is deferred to collect more packets from incoming flow. For the notation purposes, let the indicator function $\mathbb{I}(p^{*})=1$ when $p^{*}=k$; and otherwise $\mathbb{I}(p^{*})=0$, where $p^{*}=k$ corresponds to the UAV detection at time $t_{p^{*}}$.

III-C UAV Operation Mode Identification Problem

In the second task, for the detected UAV, we further identify the UAV’s operation mode. Eight UAV operation modes are labeled as $\pazocal{Z}=${“Standby”, “Hover”, “Forward”, “Backward”, “Up”, “Down”, “Right”, “Left”}. In this problem, using existing machine learning algorithms a set of multiclass classifiers are trained packet-by-packet on a real-world dataset to minimize the total operation mode misidentification cost.

IV Delay-aware UAV Detection and Operation Mode Identification

In this section, we propose a delay-aware learning-based predictive model in order to solve the problem formulated in (1) and (2). Then, we extend our work to further identify the operation mode of the detected UAV. Fig. 2 illustrates the main workflow of the proposed method.

IV-A Learning-Based Model Design

Based on the definition of traffic flow dataset $S$ in Section III-B, we further define the data of incomplete traffic flow, where only the first $j\leq n$ packets are available in the dataset, denoted as $S^{j}=\{((\mathbb{x}_{i}^{j},\boldsymbol{\tau}_{i}^{j}),y_{i}))\}_{i\in\left\{1,...,m\right\},j\in\left\{1,...,n\right\}}$ where $\mathbb{x}_{i}^{j}=(x_{i,1},x_{i,2}...,x_{i,j})$ and $\boldsymbol{\tau}_{i}^{j}=(\tau_{i,1},\tau_{i,2},...,\tau_{i,j})$ indicate the sequence of packet size and inter-arrival time of the $i$th trace in the $j$th subset, respectively. Next, for each dataset $S^{j}$, we generate a design matrix of $\mathbb{X}^{j}=[X_{1}^{j},X_{2}^{j},...,X_{m}^{j}]^{T}\in\mathbb{R}^{m\times 2l}$, where $X_{i}^{j}\in\mathbb{R}^{2l}$ is a row vector as $X_{i}^{j}=\left[V_{1}(\mathbb{x}_{i}^{j}),...,V_{l}(\mathbb{x}_{i}^{j}),V_{1}(\boldsymbol{\tau}_{i}^{j}),...,V_{l}(\boldsymbol{\tau}_{i}^{j})\right]$ where $V_{1}(\cdot),...,V_{l}(\cdot)$ are functions which compute the statistical features of the input samples (i.e., $\mathbb{x}_{i}^{j}$, $\boldsymbol{\tau}_{i}^{j}$ ). $l$ denotes the number of features. A list of statistical feature functions with their associated computation formula is shown in Table I.

Feature selection based on re-weighted $\ell_{1}$-norm by considering both feature discriminative power and computation time cost: Different features have different significance; we use $W^{j}=\{W_{1}^{j},W_{2}^{j},\cdots,W_{2l}^{j}\}\in\mathbb{R}^{2l}$ to denote the weight vector of $j$th subset for all the $2l$ features. Therefore, $W_{i}^{j}=0$ means that the $i$th feature of $j$th subset is not useful and can be discarded. Then, given the $j$th design matrix $\mathbb{X}^{j}$ and $y\in\pazocal{Y}$, our problem is to learn a predictive mapping $h_{\gamma}^{j}(X_{i}^{j},W^{j})\rightarrow y_{i}$ for $i=1,2,\cdots,m$ such that:

1. the disagreement between $h_{\gamma}^{j}(X_{i}^{j},W^{j})$ and $y_{i}$ is minimized; and
2. the number of non-zeros in $W^{j}$ is minimized.

Other than the fact that different features have different discriminative power for identification, we also observe that different features consume different amounts of computation time (refer to Table I). Therefore, in the $j$th subset among the features that bring the same discriminative power in UAV identification, we tend to remove the one(s) that consume more time. This requires us to give personalized penalty on each different feature. The more time one consumes, the more penalty it is given. Thus, instead of using conventional $\ell_{1}$-norm regularization [34] that penalizes all the features evenly, we propose the following new objective loss function with the re-weighted $\ell_{1}$-norm:

[TABLE]

where the strength of penalty $\lambda_{i}^{j}$ for $i$th feature in $j$th subset is proportional to the computational time for this feature. Therefore, the objective function in (3) will minimize the misclassification error and enforce some $W_{i}^{j}$’s to be zeros, especially those that consume more computation time. Since for computing of expected misclassification cost function, we shall need probabilistic output of the classifier, then we choose $h_{\gamma}^{j}(\cdot)$ to be one-vs-all logistic regression function [35]. One-versus-all logistic regression is a generalized version of the logistic regression into multiclass classification.

Next, we compute the training expected misclassification cost function on each trace $i$ for every subset $j$ as follows:

[TABLE]

where $P(y_{i}|X_{i}^{j})=1$ if $\hat{y}=y_{i}$ and [math] otherwise. Then, a set of one-vs-all logistic regression classifiers $\mathcal{H}=\left\{h_{\gamma}^{j}\right\}$ for $j=1,...,n$ and $\gamma\in\left\{\text{UAV}_{1},\text{UAV}_{2},...,\text{UAV}_{1-\upsilon},\text{non-UAV}\right\}$ are trained. Based on the probabilistic output of one-vs-all logistic function, we can compute $P_{j}(\hat{y}=\gamma|X_{i}^{j};W^{j})=h_{\gamma}^{j}(X_{i}^{j},W^{j})$. $C^{j}(\hat{y}|y_{i})$ denotes the misclassification cost function of training dataset. $C^{j}(\hat{y}|y_{i})$ = 1 if $\hat{y}=y_{i}$ and [math] otherwise, and $\hat{y}=\max\limits_{\gamma}h_{\gamma}^{j}(X_{i}^{j},W^{j})$. We summarize the model training phase for UAV identification in Algorithm 1.

IV-B Delay-aware Predictive Model

IV-B1 Expected missclassification cost function $C_{1}$

In the prediction phase of the incoming flow $\mathbb{\tilde{x}}({t_{k}})$, in order to compute the expected misclassification cost function $C_{1}$, $E^{j}(X_{i}^{j})$ is weighted based on the incoming traffic’s Euclidean distance from every trace in the training dataset. Consider $\mathbb{\tilde{x}}(t_{k})$ be the incoming flow and $\tilde{X}^{k}\in\mathbb{R}^{2l}$ its corresponding feature values. The weight function is defined as a normalized sigmoid function by $f_{w_{i}}^{k}=s_{i}^{k}/\sum_{i}^{m}s_{i}^{k}$ where $s_{i}^{k}=1/1+exp^{-\eta\Delta_{i}^{k}}$, and $\eta$ is some positive constant, and $\Delta_{i}^{k}={\bar{D}_{i}-d_{i}^{k}}/\bar{D}_{i}$ is the normalized average distances between $\tilde{X}^{k}$ and all the traces in the training dataset [16]. $d_{i}^{k}=||\tilde{X}^{k}-X_{i}^{k}||_{2}$ indicates the Euclidean distance of the incoming flow from $i$th trace in the dataset. In fact, the weight function $f_{w_{i}}^{k}$ plays the role of a similarity function which measures how close the incoming traffic flow is to each of the traces in the training dataset. Hence, the expected misclassification cost function for $\mathbb{\tilde{x}}(t_{k})$ is defined as follows:

[TABLE]

The above equation indicates that more weights are multiplied to the training expected misclassification value of the $i$th trace if its distance from the incoming flow is larger and vice versa.

IV-B2 Estimated time cost function $C_{2}$

For the incoming flow $\mathbb{\tilde{x}}(t_{k})$, future packet arrival times are unknown and random. This uncertainty in packet arrival times introduces difficulties in constructing a delay-aware UAV identification algorithm. In order to tackle this challenge, we propose to estimate the incoming flow’s future packet inter-arrival time according to the exponential distribution with parameter $\hat{\mu}_{i}$ for $i=1,...,\upsilon$ achieved by MLE method [36]. In Table II, we present the goodness-of-fit statistics to show that exponential distribution provides a good approximation for packet inter-arrival time estimation. For the illustration purposes, we also graphically show the goodness-of-fit for Bebop 2 and DJI Spark in Fig. 3.

Exponential distribution parameter estimation using MLE: Let $\{\pazocal{T}_{n}^{i}\}$ be a sequence of $n$ independent and identically distributed (i.i.d.) exponential random variables. Thus, $\pazocal{T}_{j}^{i}\sim\text{Exp}(\mu_{i})$ has a probability density function (pdf) of $f_{\pazocal{T}^{i}}(\tau_{j}^{i})=\mu_{i}\text{exp}(-\mu_{i}\tau_{j}^{i})$ for $\tau_{j}^{i}\geq 0$ with parameter $\mu_{i}$, where $j=1,...,n$, $i=1,...,\upsilon$ and $\upsilon=|\pazocal{Y}|$. Given the data sequence $\{\pazocal{T}_{n}^{i}\}$, our goal is to estimate the average packet inter-arrival time (i.e., $\mu_{i}$). Since $\pazocal{T}_{j}^{i}$ for $i=1,...,\upsilon$ and $j=1,...,n$ are assumed to be i.i.d., then the likelihood function is given by

[TABLE]

By taking logarithm of both sides in (6), we obtain the log-likelihood function as

[TABLE]

Then, maximum log-likelihood estimation of $\mu_{i}$ is achieved by solving the first order maximization problem of

[TABLE]

as $\frac{d}{d\mu_{i}}\boldmath{l}(\mu_{i};\tau_{1}^{i},\tau_{2}^{i},...,\tau_{n}^{i})=0$, which results in

[TABLE]

Next, we estimate the packet inter-arrival time of the incoming traffic flow $\mathbb{\tilde{x}}(t_{k})$, through the following steps: First, the Euclidean distance between $\mathbb{\tilde{x}}(t_{k})$ and each trace in the training set is computed. Second, the class label of the trace which has a minimum distance from the incoming flow is identified. Third, the average inter-arrival time of the identified class is selected from (9) to estimate the packet inter-arrival time of $\mathbb{\tilde{x}}(t_{k})$ using the exponential distribution.

Now, let $\tilde{\tau}_{i+1}=t_{i+1}-t_{i}$ for $i=k,...,n$ be the packet inter-arrival time of the $\mathbb{\tilde{x}}(t_{k})$ estimated by the above steps. Then, the estimated time cost function is obtained as

[TABLE]

where $C_{2}(t_{p})$ is a strictly increasing function.

IV-B3 MLE performance metric

We use MSE metric to measure the performance of the parameters estimated by the MLE. Considering an incoming traffic flow $\mathbb{\tilde{x}}(t_{k})$ and letting $\tau_{i+1}$ for $i=k,...,n$ be the true packet inter-arrival time of $\mathbb{\tilde{x}}(t_{k})$, we have

[TABLE]

where $MSE_{p}$ denotes the MSE estimation of packet inter-arrival time of $\mathbb{\tilde{x}}(t_{k})$ when $p$th packet arrives.

IV-B4 Estimated expected total cost function $J$

According to (1), the total cost function $J$ is defined based on $C_{1}$ and $C_{2}$ which can be computed using (5) and (10), respectively. Algorithm 2 summarizes the total cost function estimation and incoming traffic flow’s identification phase.

IV-C UAV Operation Mode Identification

We identify eight common operation modes for consumer UAVs in the market which are labeled as $\pazocal{Z}=${“Standby”, “Hover”, “Forward”, “Backward”, “Up”, “Down”, “Right”, “Left”}. UAVs’ operation mode is based on the type of the command they receive from the controller. Each UAV operation mode produces a distinct traffic pattern in the Wi-Fi network. This pattern depends on the type of the command issued by the controller which governs different packet size and inter-arrival time in the trace. Therefore, a multiclass classification model trained on a suitable dataset can identify a UAV’s operation mode. Given a dataset which contains a specific UAV’s Wi-Fi traffic traces labeled with the operation modes mentioned in set $\pazocal{Z}$, two well-recognized multiclass classification algorithms, SVM and random forest (RF) are applied to create the discriminative model. Then, the incoming traffic flow $\mathbb{\tilde{x}}({t_{k}})$ is provided as an input to the corresponding multiclass classifier to identify the operation mode of the detected UAV.

V Data Collection and Preparation

V-A UAV Detection Dataset

We collect traffic flows from eight types of consumer UAVs shown in Fig. 4: Parrot Bebop 1 Quadcopter Drone (Bebop 1), Parrot Bebop 2 Quadcopter Drone (Bebop 2), DJI Spark (Spark), DBPower UDI U842 Predator FPV (UDI), DBPOWER Discovery FPV (Discovery), DJI Tello (Tello), Tenergy TDR Phoenix Mini RC Quadcopter Drone (TDR), and Wingsland Mini Racing Drone (Wingsland). We use a DELL Latitude laptop embedded with a wireless network interface card (NIC), Intel Corporation Wireless 8260, operating in promiscuous mode to monitor and collect the Wi-Fi network traffic. For each UAV type, we collect the UAV traffic while they are flying and streaming video to the controller. To do so, we set the channel frequency of the monitoring sensor in the same channel as the UAV’s operating channel, then run Wireshark version 2.4.11 to capture the Wi-Fi traffic data. Each UAV type dataset contains 3,000 traffic traces with each trace having $n=200$ consecutive packets.

After collecting the data and identifying the UAVs’ traffic flows, we clean the data and prepare it for the training and testing dataset. In the data cleaning phase, we remove all the broadcast packets (e.g., 802.11 beacon frames), damaged packets and packets with only receiving address (e.g., 802.11 ACK frames). The remaining packets include video streaming, control commands, UAV’s response to the control commands, and UAV status updates such as direction, velocity, height and GPS information. In Fig. 5, we show the packet size distribution of the UAV types used in our experiment.

Note that the data cleaning is performed in order to train and test the classification model offline. However, when testing in the real scenario the incoming traffic may consist of broadcast packets, or packets with only ACK frames as well which all are easily discarded by the predefined filtering option on the packet capturing/monitoring software (i.e., Wireshark) before entering to the delay-aware UAV detection system.

V-B Non-UAV Dataset

In an effort to make a diverse non-UAV dataset, we create a dataset which consists of two main sub-dataset: First, we use the Wi-Fi data traffic available online from CRAWDAD database [37]. We choose this dataset because of the following reasons.

1. This dataset consists of live and non-live video streaming traffic captured from commonly seen popular applications such as Google Hangouts, ooVoo, Skype, TED and Youtube.
2. The traffic data are collected from a smartphone app where the user makes a diverse set of mobility patterns. Second, we have also captured encrypted Wi-Fi traffic on a university campus Wi-Fi network where a mixed multiple traffic types such as video streaming, social network apps, VoIP, email, web browsing applications are usually running. If the UAV identification system is set up on the campus, our method should be able to differentiate UAV traffic from these non-UAV traffic. The non-UAV dataset (Google Hangouts, ooVoo, Skype, TED, Youtube, and Campus traffic) also contains 3,000 traffic traces with $n=200$ consecutive packets.

V-C UAV Operation Mode Dataset

The following steps are taken for an operation mode data traffic collection of a specific UAV type:

1. Wi-Fi connection is established between the UAV and controller.
2. A specific operation mode command (e.g., “Forward”) is given via controller to the UAV and is held.
3. Wi-Fi medium monitoring sensor is activated to monitor the wireless channel traffic.
4. Wireshark is run on the promiscuous mode to capture the packets.
5. Before releasing the command in the controller, first, Wireshark is stopped, and then the collected traffic is saved and labeled according to the commanded operation mode.
6. This process is repeated for all the operation modes until enough data traffic is collected.

VI Performance Evaluation

VI-A Learning-based Model Performance Evaluation

By randomly sampling the dataset, we split the whole dataset into training and testing datasets with the ratio of $70\%$ and $30\%$, respectively. We create subset $S^{j}$ for $j=5,...,200$ by adding packet-by-packet information to each subset $j$ according to the step 2 in Algorithm 1. Then, we form the design matrix $\mathbb{X}^{j}$ for $j=5,...,200$ by extracting $2l=24$ statistical feature values listed in Table I. One-vs-all logistic regression multiclass classification algorithm with re-weighted $\ell_{1}$-norm technique proposed in (3) is run over each design matrix $\mathbb{X}^{j}$.

Fig. 6(a) illustrates the accuracy of the classification algorithm in training and testing on each design matrix $\mathbb{X}^{j}$. The shaded areas denote the regions surrounded by one standard deviation above and below the mean accuracy. The results show that a mean testing accuracy of higher than 88% is achieved when the information of fifty or more packets ($j>50$) are available in the subset. The model learning process also confirms the intuition that as more consecutive packets are available in the subset, the mean accuracy of predictive model increases. F-measure (weighted harmonic mean of precision and recall) for different UAV types and non-UAV is shown in Fig. 6(b). Average F-measure of higher than 86% is achieved on the test data for $n=200$. Fig. 7 also shows the corresponding precision and recall which indicate an acceptable discriminative power of the trained classifiers.

As a closely similar and related multiclass classification algorithm to one-vs-all logistic regression, we apply linear discriminant analysis (LDA) statistical method on the dataset [35]. This method is a generalized version of statistical Fisher’s LDA. The LDA method finds a linear combination of the features to distinguish different classes in the dataset. The output of this analysis is shown in Fig. 6(c) for $n=200$. Various types of linear feature combination for different classes are shown in this figure with arrows followed by its associated feature index ($\mathbin{\leavevmode\hbox to4.22pt{\vbox to6.42pt{\pgfpicture\makeatletter\hbox{\hskip 0.27986pt\lower-0.27986pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{}\pgfsys@setlinewidth{0.55972pt}\pgfsys@invoke{ }{}{{}}{} {}{}{}{{}}{} {}{}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{3.65973pt}{5.85558pt}\pgfsys@moveto{0.0pt}{5.85558pt}\pgfsys@lineto{3.65973pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}$1, $\mathbin{\leavevmode\hbox to4.22pt{\vbox to6.42pt{\pgfpicture\makeatletter\hbox{\hskip 0.27986pt\lower-0.27986pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{}\pgfsys@setlinewidth{0.55972pt}\pgfsys@invoke{ }{}{{}}{} {}{}{}{{}}{} {}{}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{3.65973pt}{5.85558pt}\pgfsys@moveto{0.0pt}{5.85558pt}\pgfsys@lineto{3.65973pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}$2, … , $\mathbin{\leavevmode\hbox to4.22pt{\vbox to6.42pt{\pgfpicture\makeatletter\hbox{\hskip 0.27986pt\lower-0.27986pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{}\pgfsys@setlinewidth{0.55972pt}\pgfsys@invoke{ }{}{{}}{} {}{}{}{{}}{} {}{}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{3.65973pt}{5.85558pt}\pgfsys@moveto{0.0pt}{5.85558pt}\pgfsys@lineto{3.65973pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}$24). Successive discriminant function in the LDA analysis provides four proportions $LD1=0.70,LD2=0.1448,LD3=0.0480$, and $LD4=0.0232$, which describes the proportion of between-class variances. It is well visualized in this figure that how UAV types and non-UAVs are distinguished on $LD2$ verse $LD1$ as a result of the features’ linear combination.

VI-B Feature Selection and Computation Time

Our objective function in (3) jointly minimizes the missclassification error and runtime by discarding useless features. Fig. 8(a) shows the set of selected features for each model trained on $j$th subset for $j=5,...,200$. As it is shown in the figure, when the sample size is small (e.g., $n<30$), all the features are selected by the model. This is because, on the one hand, small sample size does not provide enough information to the classifier to distinguish different classes with high accuracy, and on the other hand, it consumes less amount of time to compute the feature values. However, as the sample size increases, misclassification error is reduced and the feature computation time increases which results in the smaller set of selected features by the algorithm.

In order to evaluate the impact of feature selection method on the prediction time, we select 1,000 traces uniformly at random from the UAV dataset and consider them as incoming flows (i.e., $\mathbb{\tilde{x}}(t_{k})$). Then, we compute the feature generation time of the flows for $k=5,...,200$. Fig. 8(b) illustrates the mean total feature generation time of the flows versus the number of the packets in the trace with and without feature selection method. The shaded area denotes one standard deviation above and below the mean total computation time. The results show that as the number of the packets increases, with feature selection, the mean total feature generation time oscillates in a non-increasing trend depending on the number of selected features. However, without feature selection, the total computation time increases when the number of packets increases. Therefore, the proposed feature selection method reduces the prediction runtime despite the fact that the sample size is increasing.

VI-C MLE Performance Evaluation

For each UAV type, we select $1,000$ traces uniformly at random from the UAV dataset. We consider the selected traces as incoming traffic flows. Then, we follow the step 2 in Algorithm 2 to estimate the packet inter-arrival time of each flow. Using (11), we evaluate the MLE-based estimation performance. Fig.  8(c) shows the mean MSE between the true and estimated packet inter-arrival time with shaded area of one standard deviation. The results show that as more packets arrive, the mean MSE decreases. This means that as more packets are captured, the cost function $C_{2}$ estimation improves as the estimation accuracy of inter-arrival time enhances. This results in achieving high quality delay-aware UAV identification.

VI-D Delay-aware UAV Identification Test

We consider eight types of incoming UAV traffic flows each belonging to a specific UAV type, and run the delay-aware UAV detection algorithm on them. Due to the space limitation, we only show the test results for Bebop 1 in three steps Fig. 9(a), (b), (c), and summarize the outcome in the far right table in Fig. 9(d).

In Fig. 9(a), $k=10$th packet arrives at time $t_{k}=20.56ms$ and based on the received traffic flow till then, total cost function $J$ is estimated. In this case, it is estimated that the minimum total cost function will occur when $p^{*}=71$th packet arrives at $t_{p^{*}}=139.41ms$. Therefore, the decision for the flow detection is deferred. The far right column of the table in Fig. 9(d) indicates that if the UAV identification is performed in $k=10$, then the detection probability will be $42.54\%$ ($Pr=0.4254$). In Fig. 9(b), $k=40$th packet arrives at time $t_{k}=84.75ms$. In this case, the algorithm estimates that the minimum expected total cost function will occur when $p^{*}=74$th packet arrives at $t_{p^{*}}=142.94ms$. If the identification is performed in $k=40$, then with the probability of $Pr=0.6891$ the flow will be detected as a Bebop 1 traffic flow. This process is continued until the arrival of the $k$th packet for which $k=p^{*}$. According to Fig. 9(c), this condition is satisfied when $k=75$th packet arrives at $t_{k}=146.42ms$ for which $k=p^{*}=75$. In this case, UAV detection is performed and the detection probability is $Pr=0.9015$.

Next, we select 1,000 traffic traces uniformly at random from the test dataset and test the flows based on the proposed delay-aware UAV early detection algorithm. Table III shows the test results where $E[p^{*}]=\frac{1}{N_{\gamma}}\sum_{i=1}^{N_{\gamma}}p_{i}^{*}$ denotes the average optimal number of packets, $E[t_{p^{*}}]=\frac{1}{N_{\gamma}}\sum_{i=1}^{N_{\gamma}}t_{p_{i}^{*}}$ indicates the average arrival time of $p^{*}$th packet where $N_{\gamma}$ is the number of selected traces for class $\gamma$ and $\gamma\in$ $\left\{\text{Bebop 1},\text{Bebop 2},\text{Spark},\text{UDI},\text{Discovery},\text{Tello},\text{TDR},\text{Wingsland}\right\}$. In Table III, accuracy is defined as the number of correct detection divided by the total number of traces selected for the test. The results show that for the eight tested UAV types, our proposed method can detect and identify the UAVs in average within $0.15-0.35s$ with high average accuracy of $85.7-95.2\%$.

VI-E UAV Detection Distance

UAV detection range is quite dependent on the Wi-Fi traffic monitoring sensor’s hardware specification (i.e., antenna type and gain). In this experiment, we have used a DELL Latitude laptop embedded with a wireless network interface card (NIC), Intel Corporation Wireless 8260, operating in promiscuous mode to monitor and collect the Wi-Fi network traffic. Considering this type of packet capturing system, for our experiment shown in Fig. 10, in the line-of-sight (LoS) and non-line-of-sight (NLoS) i.e., blocked by a wall/trees, the system can detect the introducing UAV in the range of 70m and 40m, respectively. For the distances beyond these ranges due to heavy packet loss the detection accuracy reduces significantly. This will be our next challenging problem to tackle the UAV detection using traffic identification when the traffic suffers from packet loss.

VI-F UAV Operation Mode Identification Evaluation

Consumer UAVs’ operation mode capabilities maybe different from each other depending on the vendor specifications and manufacturing model. Here, for the UAV types, Bebop 1, Bebop 2 and DJI we identify eight common and popular operation modes as $\pazocal{Z}=${“Standby”, “Hover”, “Forward”, “Backward”, “Up”, “Down”, “Right”, “Left”}. However, FPV does not support the “Hover” mode, so we exclude this mode from set $\pazocal{Z}$ for this type of UAV.

In order to identify the operation mode of these UAVs, we apply SVM and RF multiclass classifiers on the collected real-world data traffic. For each UAV type, we train the SVM and RF predictive model packet-by-packet for $n=10,...,300$ by tuning the best model parameters for each subset. For the SVM classification method, we utilize radial basis function (RBF) kernel and tune the best model parameters. For Bebop $1$, Bebop $2$, DJI, and FPV the total number of operation mode traffic traces in the training dataset is equal to $9600$, $9600$, $9600$, and $8400$, respectively. By randomly sampling each dataset, we split the whole dataset into training, cross validation and test datasets with the ratio of $60\%$, $20\%$ and $20\%$, respectively. Using $10$-fold cross validation repeated three times the best model parameters are tuned. For example, for Bebop 1’s operation mode identification when $n=300$, the best model tuned parameters are $C=64$ and $\epsilon=0.15$ with the number of support vector machines of $\{173,139,142,131,126,174,150,142\}$ for each operation mode in set $\pazocal{Z}$, respectively.

Fig. 11 illustrates the accuracy of the classification in training and testing for the four UAV types when SVM and RF are utilized for operation mode identification. The gray (darker) and green (lighter) lines denote the mean accuracy of training and testing with the shaded area of one standard deviation, respectively. Since the SVM and RF models are trained and cross validated for parameter tuning on different UAV types operation mode dataset, training and testing accuracy varies to some extent for each UAV type. However, both SVM and RF methods show an acceptable accuracy in effectively distinguishing the tested UAVs’ operation modes.

Fig. 12 illustrates the results of the feature importance analysis for various number of packets in the set. We can see that for different UAVs, the most important feature sets can be different. For example, for Bebop 1 operation mode identification, $V_{5}(x)$, $V_{12}(x)$ (skewness and MAD of packet size), $V_{17}(\tau)$, and $V_{24}(\tau)$ (skewness and MAD of packet inter-arrival time) are indicating high importance value. However, for Bebop 2, $V_{4}(x)$, $V_{11}(x)$ (STD and PS of packet size), $V_{21}(\tau)$, $V_{22}(\tau)$, $V_{24}(\tau)$ (Mean Square, RMS and MAD of packet inter-arrival time) are showing high importance value. This indicates that: 1) For any given UAV type the data traffic patterns for various operation modes are different. 2) The operation modes of each UAV type follows a different data traffic pattern than the other UAV types. 3) In order to train an effective model for the UAV operation mode identification, it is safe to consider all of statistical features so that the model can freely choose an effective set of important features which provides higher discriminative power.

Confusion matrices for operation mode identification of four tested UAVs when $n=300$ is shown in Fig. 13. This figure indicates the overall performance of the SVM and RF multiclass classification algorithms. In each confusion matrix, the diagonal and off-diagonal cells correspond to operation modes that are correctly and incorrectly identified, respectively. The right most column of the matrix indicates the precision (positive predictive value) and false discovery rate, as the top and bottom values of each cell, respectively. Similarly, the bottom row of the matrix shows the recall (true positive rate) and false negative rate, in top and bottom part of each cell, respectively. Lastly, the cell in the most bottom right of the matrix, indicates the overall operation modes identification accuracy and error, respectively. As the results show, the operation modes of the UAVs can be accurately identified with high accuracy of $88.5-98.2\%$ through wireless traffic fingerprinting.

VII Discussion

VII-A Significance of UAV Early Detection

Considering that a consumer UAV can fly at 50-70mph and some racing UAVs could even fly above 150mph, a delay of one second will translate to a flying distance of 22m to 66m, which can be significant in practice for incident responses and safety/privacy protection. Therefore, reducing the detection delay is paramount important in the UAV invasion detection application. Another improvement in detection time can be achieved using features’ computational dependencies property [38, 39]. Statistical features shown in Table I are computationally dependent, so new techniques proposed in [38] can be applied to reduce the detection time. If number of the features are large, then large-scale feature computational dependency graph proposed in [39] can be employed to reduce the detection time as much as possible.

VII-B Applicability to Other Communication Protocols

This work is based on the observation that many of consumer UAVs utilize Wi-Fi communication protocol for remote pilot control and video streaming. However, some type of consumer UAVs (e.g., DJI Phantom, 3DR Solo, Yuneec) may use other types of custom communication protocols such as Lightbridge, Sololink and Yuneec protocol. Our proposed framework is applicable to other types of consumer UAVs that use different communication protocols. Our framework works for encrypted wireless traffic and only needs packet size and inter-arrival time information (no need to use any packet content information). As long as we can obtain this information, our framework can be applied to not only detect the UAV using the proposed delay-aware mechanism, but also identify its operation mode.

In this paper, the hypothesis is that UAVs present unique traffic patterns that can be separated from other non-UAV traffic due to their use of a different set of communication protocols and physical operation. We believe smart IP camera and handheld smartphone gimbal using a different set of communication protocols will be separable from UAV traffic as well.

VII-C Recognizing New Types of UAVs

In this paper, we applied supervised learning frameworks which can classify the known classes (UAVs) appeared in the training set. It will be interesting to extend our work to recognize new types of UAVs (unseen classes). It belongs to the open set recognition problem which is still an open research problem in machine learning areas. Existing technologies including [40] could be explored to recognize new types of UAVs and at the same time reducing the model retraining overhead. Although only a limited number of types of UAVs are tested in this work, the proposed framework should be able to handle a large dataset of different UAV subtypes. Through experiments, this paper has demonstrated the discriminative power of the proposed classifier, which indicates the effectiveness of proposed methodologies. The users are free to adjust dataset to cover different applications on different UAV subtypes. Through experiments, this paper has demonstrated the discriminative power of the proposed classifier, which indicates the effectiveness of proposed methodologies. The users are free to adjust both UAV and non-UAV dataset to cover different application scenarios (e.g., university campus, government building, airport, etc).

VII-D More Sophisticated Scenarios

The framework developed in this paper could be extended to tackle more sophisticated scenarios, such as simultaneous detection of UAVs operating on multiple channels. A more powerful adversary could even hop among different channels to escape from detection. Some multi-channel network monitoring mechanisms [41] could be integrated in this scenario. An intelligent adversary could change its traffic pattern by injecting packets to avoid being detected. However, this kind of adversary could be limited by energy budget (i.e., limited number of packets can be injected due to limited battery capacity) and mission requirement (i.e., genuine command control packets and video streaming packets cannot be suppressed). Therefore, an enhanced machine learning model which can effectively test sub-traffic could still be effective when facing such an intelligent adversary. Furthermore, investigating the possibility of combining traffic information and physical layer information (such as RSS and modulation schemes) to enhance the identification performance and enable UAV localization is of great interest.

VIII Conclusions

Detecting and identifying consumer UAVs is of utmost importance for regulation enforcement, forensics investigation, public security, and personal privacy protection. To complement existing physical detection mechanisms, we proposed a delay-aware machine learning-based UAV detection and operation mode identification framework over encrypted Wi-Fi UAV traffic. This framework extracts features from packet size and inter-arrival time and in the model training phase adopts re-weighted $\ell_{1}$-norm regularization with consideration of computation time among various features. Therefore, feature selection and performance optimization are integrated in one objective function. To deal with packet inter-arrival time uncertainty when estimating the cost function, we utilized model-based MLE method to estimate the packet inter-arrival times of the incoming flow. We collected a large amount of encrypted Wi-Fi traffic of eight types of consumer UAVs and conducted extensive evaluation on the performance of our proposed methods. Experimental results show that the proposed methods can detect and identify tested UAVs within $0.15-0.35s$ with the accuracy of $85.7-95.2\%$. The UAV detection range is within the physical sensing range of 70m and 40m in the line-of-sight (LoS) and non-line-of-sight (NLoS) scenarios, respectively. The operation modes of UAVs can also be well identified with accuracy in the range of $88.5-98.2\%$. The operation mode identification reveals the cyber-physical coupling property of UAVs. Based on this coupling, we can infer information on the physical status (operation mode) of UAVs given information on their cyber part (Wi-Fi traffic data).

Although this work uses Wi-Fi traffic to detect and identify consumer UAVs, we believe the proposed machine learning-based detection framework and methodology are general enough to be applied to other cyber-physical/IoT systems using different wireless communication technologies (e.g., Bluetooth and cellular). We hope this work to shed light on the cyber-physical attack co-detection or co-defense for many other CPS/IoT systems.