Joint Activity Recognition and Indoor Localization with WiFi Fingerprints

TL;DR

This paper introduces a deep learning framework that jointly recognizes human activities and localizes indoor positions using WiFi CSI fingerprints, demonstrating effective performance with real-world data.

Contribution

The work presents a novel dual-task CNN model for simultaneous activity recognition and indoor localization using WiFi CSI fingerprints, with publicly available data and code.

Findings

Achieves high accuracy in joint activity recognition and localization

Uses standard WiFi protocol and collects extensive CSI data

Demonstrates effectiveness through experimental results

Abstract

Recent years have witnessed the rapid development in the research topic of WiFi sensing that automatically senses human with commercial WiFi devices. This work falls into two major categories, i.e., the activity recognition and the indoor localization. The former work utilizes WiFi devices to recognize human daily activities such as smoking, walking, and dancing. The latter one, indoor localization, can be used for indoor navigation, location-based services, and through-wall surveillance. The key rationale behind this type of work is that people behaviors can influence the WiFi signal propagation and introduce specific patterns into WiFi signals, called WiFi fingerprints, which can be further explored to identify human activities and locations. In this paper, we propose a novel deep learning framework for joint activity recognition and indoor localization task using WiFi Channel State Information~(CSI) fingerprints. More precisely, we develop a system running standard IEEE 802.11n WiFi protocol, and collect more than 1400 CSI fingerprints on 6 activities at 16 indoor locations. Then we propose a dual-task convolutional neural network with 1-dimensional convolutional layers for the joint task of activity recognition and indoor localization. Experimental results and ablation study show that our approach achieves good performances in this joint WiFi sensing task. Data and code have been made publicly available at https://github.com/geekfeiw/apl.

Figures

Tables

Table 1. TABLE I: Precision, recall, and F1 score of the activity recognition task.

Activity	up	down	left	right	circle	cross
Precision	0.90	0.91	0.85	0.92	0.97	0.84
Recall	0.98	0.89	0.83	0.92	0.77	0.89
F1 score	0.94	0.90	0.84	0.92	0.82	0.86

Table 2. TABLE II: Precision, recall, and F1 score of the indoor localization task.

Location	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10	#11	#12	#13	#14	#15	#16
Precision	1	0.88	1	0.87	1	1	1	1	1	1	1	0.89	0.95	0.89	1	0.85
Recall	1	0.88	1	0.76	1	1	0.94	0.94	0.94	1	1	0.94	1	0.94	1	0.94
F1 score	1	0.88	1	0.81	1	1	0.97	0.97	0.97	1	1	0.92	0.97	0.92	1	0.89

Table 3. TABLE III: Inspired by t-SNE visualization, we propose ResNet1D-[1,1,1,1]+, which outperforms ResNet1D-[1,1,1,1] on activity recognition.

Model	Activity Recognition	Indoor Localization
ResNet1D-[1,1,1,]	88.13%	95.68%
ResNet1D-[1,1,1,1]+	89.57%	95.68%

Table 4. TABLE IV: Expansible study and comparison with DTW+KNN. Deeper networks work better on indoor localization task, wheres work worse on activity task. All ResNet1Ds outperform the baseline methods, i.e., DTW+kNN and SVM  [ 55 ] . ‘AR’ and ‘IL’ are abbreviations of activity recognition and indoor localization, respectively.

Method	AR	IL	Time Cost
DTW+kNN	83.45%	95.32%	3356s
SVM-RBF[55]	40.64%	70.32%	69s
ResNet1D-[1,1,1,1]	88.13%	95.68%	161s
ResNet1D-[2,2,2,2]	87.77%	96.40%	240s
ResNet1D-[3,4,6,3]	85.17%	97.12%	412s

Equations

$$y=f(x)+x.$$

$$L=L_{activity}+\lambda L_{location}$$

$$s_i^{\prime }=\frac{e^{s_i}}{{\sum }_{j=1}^K{e}^{s_j}},i\in [1,2,...,K],$$

$$L_{activity}=-log(s_t^{\prime })$$

$$ALE=\frac{1}{N}\sum _{i=1}^N{\Vert Coor(P_i)-Coor(G{T}_i))\Vert }_2^2$$

$$AME=\frac{1}{M}\sum _{i=1}^M{\Vert Coor(P_i)-Coor(G{T}_i))\Vert }_2^2$$

Full text

\titlepgskip

=-15pt

I Introduction

Channel State Information of WiFi devices have been extensively explored for human sensing tasks such as activity recognition [1, 2, 3, 4], gesture recognition [5, 6, 7, 8], indoor localization [9, 10, 11, 12], and health-care applications [13, 14, 15, 16, 17, 18, 19]. This prosperity benefits from several special properties of WiFi, including the ubiquitous deployment of commercial WiFi devices, the robustness to lighting condition and occlusion which overcomes limitation of cameras, and the non-intrusiveness sensing which requires no user’s extra effort.

Though there is abundant work on the specific aforementioned WiFi human sensing task [20, 6, 7, 3, 4, 13, 14, 15, 16, 17, 18], little work aims at completing the joint task of activity recognition and indoor localization. Carrying out the joint task would breed numerous useful human-computer interaction applications. For example, in a smart home with Internet-of-Things (IoT) devices [21, 22], the devices could precisely response differently to the same gesture command based on user’s location. More specifically, the user can use the gesture of ‘hand down’ to turn down the television in front of her, whereas she can also use the same gesture to lower the air conditioner’s temperature when standing close to the air conditioner. To our best knowledge, MultiTrack [23] is the only work that enables indoor localization and activity recognition jointly, however it requires high-end hardware modification for ultra-wide band WiFi (over 600MHz).

The joint task can be summarized as the following two folds. (1) Recognizing activities conducted at different locations. (2) Localizing the user by the activities. However, there are two major challenges lying in the way. The first challenge is that WiFi fingerprint differs even when performing a same activity but at different locations, thus we need to look for a same representation for activities conducted at all locations. The second one is that WiFi fingerprints vary when performing different activities in one location, thus we have to explore distinguished features for each location from the fingerprint variances.

To conclude the above challenges formally, WiFi fingerprint, $W$, contains two components at the same time, activity category ($A$) and user location ($L$). We denote the WiFi fingerprint as $W(A,L)$. Joint activity recognition and indoor localization task aims to learn a function $f$, which is capable to classify activity categories ($f:W(A,L)\rightarrow A$) and to localize the user ($f:W(A,L)\rightarrow L$), simultaneously. Thus we formalize the joint task as $f:W(A,L)\rightarrow(A,L)$.

To this end, in the paper we propose a novel 1-dimensional Convolutional Neural Network (C1D) including two branches, one for activity recognition and the other for indoor localization. To date, conventional 2-dimensional Convolutional Neural Networks (C2D), which have brilliant ability to learn features from raw data, boost the development of computer vision [24, 25, 26, 27, 28, 29], robotics [30, 31, 32, 33], machinery [34, 35, 36], etc. Unlike C2D that processes 2D spatial data such as images, C1D is capable to process 1D temporal data. For temporal WiFi fingerprints, we design a C1D based on the ResNet [24] to carry out the joint task of activity recognition and indoor localization.

To evaluate our proposed approach, we implement the standard IEEE 802.11n protocol in two universal software radio peripheral (USRP) sets, Ettus N210111https://www.ettus.com/all-products/un210-kit/, where one Ettus N210 broadcasts WiFi signals and the other parses Channel State Information (CSI) fingerprints of WiFi for joint task. We define 6 hand gestures for potential human-computer interaction applications, namely, hand up, hand down, hand left, hand right, hand circle and hand cross. One volunteer repeats these activities 15 times at each location (16 locations in all) and forms a dataset with 1394 samples (after excluding the invalid data). We evaluate our proposed C1D on this dataset and present the results with several metrics such as confusion matrix, F1 scores, convolutional feature maps, etc. Experimental results show our proposed C1D achieves a very promising performance in the joint task. We summarized our contributions as follows.

1. We first propose and achieve the joint task of activity recognition and indoor localization, which enable practical user gesture control in smart homes for human-computer interaction applications.

2. We novelly view CSI fingerprints as time series with channel dimension and time dimension, apply a advanced 1-dimensional Convolutional Neural Network that sweeps along the time dimension of the CSI fingerprints, and achieve the joint task of activity recognition and indoor localization.

3. We implement IEEE 802.11n protocol in two USRP sets and build a dataset specifically for the joint task. We evaluate the performance of proposed deep network on this dataset and fully analyze the results.

II Related Work

II-A CSI Fingerprints

CSI fingerprints of WiFi have been widely utilized for activity recognition [6, 7, 3, 4, 13, 15] and indoor localization [37, 38, 39, 40]. As for activity recognition, in [3, 13, 15], CSI fingerprints are used to detect user falling especially for the elderly-care system. In [7], CSI fingerprints are used to infer user keystroke. Further in [41], researchers find that CSI fingerprints can reveal people’s typing when they use smart phones in public WiFi. In [6, 42] CSI fingerprints are designed for hand sign recognition for human-computer interactions. As for indoor localization, [37, 38, 39, 40] collect CSI fingerprints corresponding to people locations, and train classifiers to localize people with collected CSI fingerprints. To our best knowledge, there is no work on joint activity recognition and indoor localization, which is very useful in controlling different smart devices at different locations with a set of pre-defined activities. We achieve this task by a dual-branch Convolutional Neural Network.

II-B CSI Fingerprints Classification

There exist three popular approaches in CSI fingerprints classification. (1) Hand-crafted features + Support Vector Machine (SVM) [43]: [13, 15] apply statistical values of CSI time-series such as the mean, maximum, minimum, entropy, etc., as features to train SVM with kernel methods for CSI fingerprints classification. This approach requires expertise in designing features, which is even much harder on joint activity recognition and indoor localization. (2) Dynamic Time Wrapping (DTW) + k Nearest Neighbors (kNN): [3, 6, 42] first build a dataset with CSI fingerprints. When classifying a test CSI sample, this approach requires computing all distances between the test sample and all samples in the dataset, which is time-consuming compared to pre-training a classifier first. (3) Deep learning: [37, 40] utilizes deep Boltzmann Machine (DBM) to do indoor localization. However DBM relies heavily on careful design and tricks to converge. [38, 39] apply 3-5 convolutional layers on activity recognition. In general, the shortage in the depth limits the performance. [42] utilizes ResNet [24] and Inception [26] to categorize CSI fingerprints, whereas it only handles single moment CSI, i.e., rather than handling temporal CSI fingerprints. In this paper, we propose a ResNet-based Convolutional Neural Network to do CSI fingerprints classification.

II-C 1D Convolutional Neural Network

Conventional Convolutional Neural Network (C2D) [25, 26, 24] are designed for 2D inputs such as images. C2D applies 2D convolutional kernels to sweep along the width and height of an image to capture its semantic and structural information for image classification [25, 26, 24], object detection [44], instance segmentation [27], etc. In [45, 46], researchers apply 3D CNN (C3D) on video data, which sweeps along the width, height, and time of the video to capture information both in spatial and in temporal. In this paper, we apply 1D convolutional kernels to sweep along the time axis of the CSI fingerprint series to capture the temporal information of CSI fingerprints, which works well in the joint task of activity recognition and indoor localization.

\Figure

[t!](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.97]figs/apl_system_small.pdf Main hardwares: Ettus USRP N210 and Ettus Clock.

\Figure

[t!](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.48]figs/apl_system_block_2.pdf System framework. The system contains two sets of personal computers and USRPs, which work as the WiFi transmitter and receiver, respectively. An Ettus clock synchronizes the two sets.

III Data Collection

III-A Hardware

We implement the standard IEEE 802.11n protocol in two universal software radio peripherals (USRPs) to collect CSI fingerprints. As shown in FIGURE II-C, the first two figures are the top view and front view of the USRP (Etuss N201), respectively. The USRP is mainly composed of a mother board, a daughter board and a WiFi antenna, which is used to broadcast or receive WiFi signals under the control of GNU Radio222https://www.gnuradio.org/. The details are listed below. Meanwhile, the assembling diagram is shown in FIGURE II-C.

1. Etuss N210s: A hardware with field programmable gate array (FPGA) that can be embedded IEEE 802.11n protocol to send and receive WiFi packages for CSI fingerprints.

2. Etuss Clock333https://www.ettus.com/all-products/OctoClock-G/ and synchronization cables: Synchronizing N210s with GPS clock to avoid a WiFi phase shifting caused by the clock differences between two N210s.

3. Antennas: To broadcast or receive WiFi signals under the control of GNU Radio444https://www.gnuradio.org/

4. Computers and Ethernet cables: To control N210s when are set in a same local area network as N210s.

\Figure

[t!](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.48]figs/apl_doing_small.pdf Six gesture commands mainly for human-computer interaction applications in smart home, i.e., hand circle, hand up, hand cross, hand left, hand down, and hand right.

\Figure

[t!](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.48]figs/apl_location.pdf One volunteer does activities at 16 locations.

\Figure

[t!](topskip=0pt, botskip=0pt, midskip=0pt)[width=1.0]figs/apl_upsample.pdf Signal preprocessing. we manually annotate the action duration (leftmost), split CSI samples during action (middle), and upsample the splitted CSI series to be size of 192 (rightmost). (take the 29th subcarrier series when the volunteer plays a ‘circle’ action at #2 position for example.)

\Figure

[t!](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.48]figs/apl_all_65.pdf CSI fingerprint samples of 6 activities on #10 position.

\Figure

[t!](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.48]figs/apl_circle_46.pdf Do ‘circle’ at the #3 position. CSI samples vary in profiles. CSI may be partly captured because of late action start.

\Figure

[t!](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.48]figs/apl_circle_556476.pdf Do ‘circle’ at the #6 (left), the #9 (middle), and the #15 position.

III-B Activity and Location

We design 6 activities, namely, hand up, hand down, hand left, hand right, hand circle, and hand cross, for human-computer interaction applications, as shown in FIGURE III-A. This cluster of activities covers the majority of daily commands for smart Internet-of-Things home, where using cameras are not practical due to security and privacy concerns. Here we illustrate how our proposed activities work by the case of television. “Hand up” and “hand down” can be used to turn up and down the voice volume, respectively; “Hand left” and “hand right” indicate switching channels; “Hand circle” and “hand cross” are for CONFIRM command and CANCEL command, respectively.

Besides recognizing activity in smart home, localizing the user when s/he is doing an activity is also crucial for the joint task. By combining user’s activity and location together, we are able to infer user’s intention more precisely and make it possible for users to control a range of smart devices with the same activity. For example, the user may want to communicate with the television when sitting on the sofa, whereas s/he probably needs to control the air conditioner (AC) when standing in front of an AC. To make a proof-of-concept experiment, we collect CSI fingerprints when a volunteer does 6 activities (shown in FIGURE. III-A) at 16 locations in one room. We illustrate the spatial relationship in FIGURE. III-A, where the 16 locations are selected evenly with a purpose to cover most central area of the room. The USRPs are fixedly placed besides the selected locations. In all, the volunteer repeats each of the 6 activities 15 times at 16 locations and forms a dataset with 1440 samples.

III-C CSI fingerprint analysis

We visualize some samples varying in activities and locations to present the challenges of joint task of activity recognition and indoor localization. FIGURE III-A shows CSI fingerprints when the volunteer does 6 activities at #10 location, in which the x-axis is the sampling index (time) and the y-axis is the amplitude of the CSI fingerprints. There are 52 time series in each CSI fingerprints, differed with 52 colors in FIGURE III-A. 52 is the number of orthogonal frequency division multiplexing (OFDM) [47] sub-carriers that carry data in parallel in WiFi protocol. FIGURE III-A demonstrates that CSI fingerprints vary when the user conducts 6 activities at a same location.

FIGURE III-A illustrates 3 CSI fingerprint samples when the volunteer carries out “hand circle” at #3 location. FIGURE III-A shows that though the volunteer plays the same activity at the same location, CSI fingerprints are still very different in time-serial profile (left and middle), and in the start point of the activity (left and right). Besides, performing the same activity at different locations also largely varies CSI fingerprints as illustrated in FIGURE III-A, making it challenging to find out shared features for one activity at all locations.

IV Methodology

IV-A Preprocessing

As shown in FIGURE II-C, we have one transmitting antenna and one receiving antenna. As shown in FIGURE III-A, the number of sub-carriers and the streams for each CSI fingerprint are 52 and 192, respectively. This setting makes each CSI fingerprint a matrix with size of $52\times 192$. Note that we only use CSI amplitude and ignore the CSI phase for the fusion of both is out of range of this paper. Because CSI fingerprints vary according to different activity start time and finish time, we manually annotate the activity duration to prepare useful signals for further use. We take the time series of 29th sub-carrier as the visualization example to show a duration annotation in FIGURE III-A (left). This annotating process enables us to directly use the segmented CSI fingerprints for the joint task. We then upsample the segmented CSI fingerprints to make them the same size using the linear interpolation (in our experiment, the sizes of original and upsampled CSI fingerprints are both 192). One interpolated sample is shown in FIGURE III-A (right).

IV-B 1D Convolutional Neural Network

As illustrated in FIGURE III-A, FIGURE III-A and FIGURE III-A, CSI fingerprints are time series with 52 sub-carriers. We denote it as $C\in R^{52\times t}$, where $t$ is for sampled time, $R$ means Real number. Thus it is essentially to do time-serial matching that to use CSI fingerprints for the joint task of activity recognition and indoor localization. Recall that Dynamic Time Warping (DTW) [48, 49], one of the most prevailing approaches for time-serial matching, takes both the values and the shapes of the time series into consideration to compute the distance between two series. However, to categorize one test series, DTW has to compute the distances of the series to all samples in training database, which is time-consuming. Some work utilizes hand-crafted statistics characteristics of the time series, e.g., the mean, variance, and entropy, to represents the values and shapes of the time series [50, 13, 15], which requires expertise to design and extract these characteristics, i.e., not generic. Thus to achieve the joint task, we aim to propose an approach that (1) can capture the values and the shapes of the CSI fingerprints, and (2) should be time efficient. Since Convolutional Neural Network approaches boost current pattern recognition applications due to the ability of learning powerful features directly from raw data. In this paper, we apply 1-dimensional Convolutional Neural Network (C1D) on CSI fingerprints for joint task of activity recognition and indoor localization.

FIGURE IV-B (left) shows a 2-dimensional convolutional operation (C2D) for spatial data such as images, and FIGURE IV-B (right) illustrates C1D for temporal inputs such as WiFi fingerprints. For a C2D, the input size is $7\times 7$, a 2D convolutional kernel sized $3\times 3$ sweeps along the width and height of the input with the stride of 2, and it leads to a result of $3\times 3$. With this sweeping operation, spatial information of the input, such as the object location in an image, can be captured. Differing from C2D, C1D only sweeps along the time axis and captures temporal information in $C$ (including values and shapes), which is highly correlated with the user activities. Besides, consulting the widely used method that C2D takes 3 (3 for RGB color channels) as the channel for image data, in this paper, we take the 52 of $C$ as the channel of C1D operation (52 for 52 OFDM channels) for CSI fingerprints. Different to previous work that views the 2D CSI fingerprints as images with height and width, then applies C2D on the fingerprints [42, 51, 38], to our best knowledge, our approach is the first manner that views CSI fingerprints as time series with channel dimension and time dimension. More importantly, it makes advanced C1D can be directly applied on CSI fingerprints without bells and whistles.

\Figure

[t!](topskip=0pt, botskip=0pt, midskip=0pt)[width=0.48]figs/apl_1d_1.pdf The difference between the 2D convolutional operation and the 1D convolutional operation.

IV-C Network Framework

In computer vision community, ResNets [24] have been proved to be effective and advanced in many task, such as image classification [24], object detection [44], instance segmentation [27], etc. However the standard ResNets are implemented to process 2D inputs such as images with height and width, thus we re-implement a ResNet specifically for our temporal CSI fingerprints, termed as ResNet1D.

The main component of ResNet1D is the basic residual block as shown in FIGURE 1 (left). We denote the input fingerprints as $x$, and the output as $y$. With two convolutional layers, $x$ becomes $f(x)$. With a shortcut link, $x$ becomes a part of $y$. Thus the two branches make the output of the basic residual block

[TABLE]

In FIGURE 1 (right), we illustrate our implementation in details. In the $f(x)$ branch, $x$ is scanned by two C1Ds with size of $3\times 1$ (‘C1D$3\times$1’). Moreover, 1D batch normalization [52] (‘BN1D’) follows at each ‘C1D$3\times$1’, and a Rectified Linear Unit activation function follows the first ‘C1D$3\times$1’. In the shortcut branch, $x$ is processed by a C1D with the size of $1\times 1$ (‘C1D$1\times$1’) and a 1D batch normalization. The outputs of two branches, $f(x)$ and $x$, may have difference in the size, making it unavailable to do element-wise addition between $f(x)$ and $x$. Therefore, the ‘C1D$1\times$1’ in the shortcut branch is designed to make the sizes of $f(x)$ and $x$ the same.

Based on the basic residual block above, we build the ResNet1D as shown in FIGURE 2. The network takes CSI time series as input and predicts user activity and location in parallel as output. The network contains 11 C1D layers, where 9 are shared, and each sub-task has one C1D independently, whose parameters are listed in FIGURE 2. Taking the first C1D, ‘C1D$7\times 1$, 128’, as an example, the C1D is with the kernel size of $7\times 1$ and the output channel number of 128. Besides C1Ds, there is 1 max pooling operation following the first C1D and 2 average pooling operations following the last 2 C1Ds, respectively. Four residual blocks (RB1, RB2, RB3, and RB4) contribute the mainstream of the ResNet1D-[1,1,1,1]. Take the RB1 for example, it contains one basic residual block as shown in FIGURE. 1, where the first convolutional layer is a 1D convolution with kernel size of $3\times 1$ and out-channel number of 128 (C1D$3\times 1,128$). Note that we neglect the ‘BN1D and ‘ReLU’ for simplification. Since this network contains 4 residual blocks (RBs) and in each block there exists one basic residual block, we call it ResNet1D-[1,1,1,1]. Moreover, we use a fully-connected layer to predict each of the 6 activities with a separate score. The output of activity recognition is the activity with the highest score. Meanwhile we use a fully-connected layer to predict one location out of 16 locations where the activity is carried out.

It deserves mentioning that ResNet1D-[1,1,1,1] is an expansible framework by cascading multiple basic residual blocks in RBs. In the evaluation section, we will compare the performances of ResNet1D-[1,1,1,1], ResNet1D-[2,2,2,2], and ResNet1D-[3,3,6,3] (the latter two RB settings are defaults in ResNet [24]).

IV-D Loss function

The loss, $L$, that optimizes ResNet1D is the sum of two sub-tasks, activity recognition and indoor localization. We term it as follows.

[TABLE]

where $L_{activity}$ and $L_{location}$ are losses of activity recognition and indoor localization, respectively, $\lambda$ is to balance these two losses. Before computing $L_{activity}$ and $L_{location}$, we first normalize the prediction scores with SoftMax function,

[TABLE]

where $K$ is the categories of activities ($K=6$ for $L_{activity}$ and $K=16$ for $L_{location}$), $s_{i}$ and $s_{i}^{\prime}$ are the predicted score and normalized score for the i-th activity, respectively. Using (3), all prediction scores are normalized to 0-1 range.

Then we apply the Cross Entropy Loss function on the normalized score to compute $L_{activity}$ as follows.

[TABLE]

where $s_{t}^{\prime}$ means the normalized prediction score that belongs to (resulted from) t-th activity. With the same approach, indoor localization loss, $L_{location}$, can be computed. In our experiment, we assume activity recognition and indoor localization are of the same importance, thus we set $\lambda$ in (2) as 1 to optimize ResNet1D-[1,1,1,1].

IV-E Implementation

We implement ResNet1D with Pytorch 1.0.0 555https://pytorch.org/ in a desktop that is with the Window 7 OS and one Nvidia Titan Xp GPU. The network is trained for 200 epochs by Adam optimizer [53] with default settings ($\beta_{1}=0.9$, $\beta_{2}=0.999$). The mini-batch size is 128 and the initial learning rate is 0.005. The learning rate decays by 0.5 every 10 epochs. Before each epoch, all training data are shuffled.

V Evaluation

V-A Dataset

As III-B described, our dataset involves 6 hand activities, i.e., hand up, hand down, hand left, hand right, hand circle and hand cross, that one user conducts at 16 locations. At each location, each activity is repeated for 15 times. Thus we collect totally $16\times 6\times 15=1440$ samples. In IV-A, we manually discard samples with extremely late start point to ensure data quality, leading to a final dataset with 1394 samples. We select one out of every five samples to build the test set (278), and leave the remaining 1116 samples for the training set. Due to the test set evenly selected from all samples, the test condition is the same as the training condition. We then train the proposed deep networks with the training set and validate the networks on the test set.

V-B Learning curves

We display learning curves of loss and accuracy for the activity recognition and indoor localization in FIGURE 3. In the loss curve of activity recognition (1st subfigure), the training loss (blue line) decreases gradually, and reaches a relatively low state around the 50th epoch. Whereas the test loss curve (red line) wildly swings within the first 45 epochs, and gradually reaches to a steady state around the 75th epoch.

A phenomenon needs to be addressed is that though the training loss curve keeps relatively steady after the 50th epoch, the test loss still decreases when more training epochs are involved. We ascribe it to the process of shuffling training dataset before each training epoch, described in IV-E. The shuffle process makes the network, i.e., ResNet1D-[1,1,1,1], be optimized with different mini-batch samples in each epoch. After training loss curve reaching a steady condition, the shuffling process continuously generates (keeps generating) more mini-batch combinations and these combinations are continuously updating the network.

The accuracy curves of activity recognition are plotted in the 2nd sub-figure of FIGURE 3, where the training curve (blue line) reaches a steady condition around the 70th epoch, and the test curve reaches a steady condition around the 100th epoch. Similarly, we ascribe it to shuffling process as explained above. Besides, the learning curves of indoor localization are plotted in the 3rd and 4th sub-figures of FIGURE 3. Comparing the learning curves between the activity recognition and indoor localization, we find that the task of indoor localization converges faster and achieves a better performance.

V-C Quantitative results

We demonstrate the quantitative results including confusion matrix, prediction accuracy, precision, recall, and F1 scores in the following section.

The confusion matrix of ResNet1D-[1,1,1,1] on activity recognition and indoor localization are shown in FIGURE 4 and FIGURE 5, respectively. As shown in the two figures, we achieve a accuracy of 88.13% for activity recognition and 95.68% for indoor localization. The gap between two accuracies accords with learning curves. For activity recognition, the majority of mis-predictions happen at recognizing the gesture of hand cross. Precisely, ResNet1D-[1,1,1,1] wrongly predicts 8% of hand cross to hand left, and wrongly predicts 6% of hand cross to hand circle. Meanwhile for indoor localization, a major error is wrongly predicting 15% of #16 location as #4 location. Nevertheless, ResNet1D-[1,1,1,1] generally works well on both activity recognition and indoor localization.

We further compute the precision, recall, and F1 score from the confusion matrix, and list the results in Table I and Table II. There exists a big gap between precision and recall for the activity of hand circle. A precision of 0.97 means ResNet1D-[1,1,1,1] effectively figures out (recognize) the hand circle activity, while a recall of 0.77 indicates that ResNet1D-[1,1,1,1] tends to categorize other activities into hand circle, decreasing the F1 score of hand circle to 0.82. Besides in Table II, the lowest F1 score is on #4 location prediction, 0.81, due to the low recall. In general, ResNet1D-[1,1,1,1] achieves very promising performances.

In addition, we computer average localization error (ALE) in terms of meters by transferring localization labels via

[TABLE]

where $i$ and $N$ are the index and the number of test sample, respectively, ($N=278$); $Coor(P_{i})$ is to transfer the predicted location label to the 2D spatial coordinates according to FIGURE. III-A; $Coor(GT_{i})$ is to transfer the ground-truth label to the 2D spatial coordinates; $\left\|\cdot\right\|_{2}^{2}$ is to compute the L2 distance. Via Equation. 6, the average localization error of ResNet1D-[1,1,1,1] on our dataset is 0.0904$m$, which is accurate enough considering typical smart device distribution in homes.

Besides, we would like to report the average mis-classified error between the prediction the the ground-truth, once one sample is incorrectly predicted, via

[TABLE]

where $M$ is the number of samples that wrongly classified ($M=12$); other symbols keep the same meanings as Equation. 6. Then $AWE$ is 2.0943$m$, between two and three experimental locations in FIGURE. III-A.

V-D Data Visualization

We visualize the test set by t-SNE [54] to explore the behaviors of ResNet1D-[1,1,1,1] on the joint task. Taking the activity recognition task as an example (FIGURE 6), we reduce the input into 2 dimensions data by a t-SNE tool666https://lvdmaaten.github.io/tsne/ and display the 2-d data in the figure. For an input test sample, the reducing procedure is as follows. As IV-B said, one original CSI fingerprint $C\in R^{52\times t}$, where $t$ is 192 after the cutting and linear interpolating preprocess, described in IV-A. However t-SNE requires the input to be a long 1D vector, thus we reshape $C$ to be a vector, making a $C^{\prime}\in R^{1\times 9984}$ ($192\times 52=9984$). In addition, we repeat the reshaping over all test samples and finally visualize the reshaped samples on the 1st sub-figure in FIGURE 6, marked with the green box. We can see that the inputs are highly disordered in term of activity recognition.

Besides the raw inputs visualization, we also visualize feature maps produced by ResNet1D-[1,1,1,1] in multiple layers (FIGURE 2), i.e., feature maps after max pooling layer, RB1, RB2, RB3, RB4, feature maps before FC (the 7th sub-figure), and feature maps after FC (outputs, the 8th-subfigure). We reshape all feature maps to 1D long vectors with the same approach used for visualizing the raw inputs. FIGURE 6 shows that ResNet1D-[1,1,1,1] gradually increases the discriminative power of feature maps for the activity recognition task step by step, making classification more accurate in the deeper layers of the network. Finally in the outputs (the last sub-figure of FIGURE 6), features are learned to be effective for activity recognition.

With the similar approach, we visualize the raw inputs and feature maps after multiple layers of ResNet1D-[1,1,1,1] for indoor localization in FIGURE 7. It demonstrates that the network can effectively learn features for indoor location. In FIGURE 6, ResNet1D-[1,1,1,1] generates discriminative features after FC, whereas in FIGURE 7 it generates discriminative features beginning at RB4. Moreover ResNet1D-[1,1,1,1] is able to generate better features for the indoor localization than the activity recognition because the class clusters are more tighter compared the last sub-figure of FIGURE 6 and the last sub-figure of FIGURE 7.

More importantly in the activity recognition, we find the features are largely enhanced through its own branch because the feature before FC (7th) is much better than the features after the shared RB4 (6th). Under this consideration, we just add one more ‘C1D$3\times 1$, 512’ between the ‘C1D$3\times 1$, 512’ and the ‘AvgPool1D$4\times 1$’, named ResNet1D-[1,1,1,1]+. We train ResNet1D-[1,1,1,1]+, and find it with better performance on activity recognition, listed in Table III.

V-E Expansible study and Baselines

ResNet1D is expansible by simply customizing the number of residual block in each RBs, shown in FIGURE 2. Following the default settings in ResNet [24], we evaluate the accuracy of ResNet1D-[2,2,2,2] and ResNet1D-[3,4,6,3]. As listed in Table IV, all ResNet1Ds work well in joint activity recognition and indoor localization. Meanwhile it deserves mentioned that deeper ResNet1Ds tend to work better on indoor location, whereas work worse on activity recognition.

Besides in Table IV, we show the comparison between ResNet1Ds and two baseline methods, Dynamic Time Warping777https://www.mathworks.com/matlabcentral/fileexchange/43156-dynamic-time-warping-dtw?focused=3846333&tab=function (DTW)+kNN [48, 49], and Support Vector Machine (SVM) [55] with radial basis function kernel (RBF). All our proposed ResNet1Ds outperform the baseline DTW+kNN, and SVM-RBF. We also record the time cost of these methods, which are the sum of time cost in training and testing. SVM-RBF costs least but performs worst. DTW+kNN is a very strong baseline in time-serial classification, however it is time-consuming888Code is available at https://github.com/geekfeiw/apl.

VI Conclusion

In this paper, we propose novel 1D Convolutional Neural Networks (C1D) with two branches for the joint task of activity recognition and indoor localization with WiFi fingerprints. To evaluate the proposed network, we implement IEEE 802.11n protocol in a software-defined-radio hardware, Etuss N210, collect a dataset mainly for human-interaction applications, and fully discuss the results in various aspects. Experiment results show that our proposed network can achieve joint activity recognition and indoor localization well, e.g., 88.13% on activity recognition and 95.68% on indoor localization, outperforming Dynamic Time Warping and Support Vector Machine. The computational time cost is larger than Support Vector Machine, but can be reduced by training with the larger batch size. Further, we envision C1D to be one of leading approaches in CSI fingerprint processing, to this end, we release our data and code.