Real Time 3D Indoor Human Image Capturing Based on FMCW Radar

TL;DR

This paper presents a real-time, privacy-preserving 3D indoor human imaging system using FMCW radar, capable of capturing clear human images and activities without wearable sensors or privacy concerns.

Contribution

It introduces a novel FMCW radar-based method with data preprocessing, signal processing, and deep learning filtering to achieve real-time 3D human imaging in indoor environments.

Findings

Effective background static reflection removal

High-quality 3D human image reconstruction

Real-time recognition of human activities

Abstract

Most smart systems such as smart home and smart health response to human's locations and activities. However, traditional solutions are either require wearable sensors or lead to leaking privacy. This work proposes an ambient radar solution which is a real-time, privacy secure and dark surroundings resistant system. In this solution, we use a low power, Frequency-Modulated Continuous Wave (FMCW) radar array to capture the reflected signals and then construct to 3D image frames. This solution designs $1)$a data preprocessing mechanism to remove background static reflection, $2)$a signal processing mechanism to transfer received complex radar signals to a matrix contains spacial information, and $3)$ a Deep Learning scheme to filter broken frame which caused by the rough surface of human's body. This solution has been extensively evaluated in a research area and captures real-time human images that are recognizable for specific activities. Our results show that the indoor capturing is clear to be recognized frame by frame compares to camera recorded video.

Equations

$$R=\frac{c|△t|}{2}=\frac{c|△f|}{2(df/dt)}$$

$$\text{vspace}-0.0in\begin{array}{rl}& sizeX=range(R)/res(R)\\ & sizeY=range(\theta )/res(\theta )\\ & sizeZ=range(\varphi )/res(\varphi )\end{array}$$

$$s_t=A_t{e}^{-j2\pi \frac{r}{\lambda }t}$$

$$\begin{array}{rl}& s_{n,t}=\sum _{k=1}^K{A}_{n,t}{e}^{-j2\pi \frac{r_{n,k}}{\lambda }t}\\ & r_{n,k}=travel(p_k,a_n)\end{array}$$

$$P(\varphi )=|\sum _{n=1}^N{s}_{n,t}{e}^{-j2\pi \frac{nd\sin \varphi }{\lambda }}|$$

$$P(r_k)=|\sum _{n=1}^N\sum _{t=1}^T{s}_{n,t}{e}^{-j2\pi \frac{v{r}_{n,k}}{c}t}|$$

$$\begin{array}{rl}& P(r_k,\theta ,\varphi )=\\ & |\sum _{m=1}^M\sum _{n=1}^N\sum _{t=1}^T{s}_{n,m,t}{e}^{-j2\pi \frac{v{r}_k}{c}}{e}^{j\frac{2\pi }{\lambda }\cos \theta (n{d}_y\sin \varphi +m{d}_x\cos \varphi )}|\end{array}$$

$$loss(x,label)=-log(\frac{e^{x[label]}}{{\sum }_j{e}^{x[j]}})$$

Full text

I Introduction

Indoor human image capturing is of utmost important to many intelligent devices or systems. For example, robots need real-time human images to plan and change the route, and smart health system needs human images to recognize their activities thus alert when children or elderly people fall. However, most human image capturing solutions based on cameras, which makes users concern about privacy leaking problem[1, 2]. Hence, human image capturing without computer vision technology has been a very popular research topic.

While traditional camera-based solutions result in privacy issues, the wearable sensors are devised to collect and process human motion data in many smart home scenarios. However, those wearable devices are inconvenient to users because they have to remember to equip these sensors when wakeup, and take off it when typing, wash hands or strenuous exercise[3]. The worse thing is when they take off those wearable sensors, and the capture process would keep still; thus the results are unreliable at that time. Hence then, it is highly demanded to design a passive, non-invasive and real-time indoor human image capture solution so that privacy protection and convenience issue can be guaranteed. There are two more benefits to use radar or Radio Frequency (RF) technology to capture indoor human activities. One is RF solution can ”see” human in dark light condition, while the other one is RF signals can sense human activities through the wall.

In this paper, we propose a scheme of Human Image Capturing based on FMCW Radar sensors, HICFR. This work uses a FMCW radar to sense environment, then convert raw signals to 3D human images which contain spacial location information of target in real-time. It has the following highlights:

$\bullet$ It uses antenna array combined with FMCW radar to send and receive directional beamforming to sense 3D environment, the frequency is 3.3GHz to 10GHz, and it is a very low power system, with average transmit power is below -40dbm/MHz.

$\bullet$ It has data clean feature, and the proposed calibration algorithm can record static environment response, then remove them from raw signals, thus only useful human motion responses can be reserved, which makes visualization results more clear.

$\bullet$ It uses deep learning algorithm to process raw 3D images, and the deep learning model is trained to recognize whether the current frame is caused by irregular reflections, if true, filter them out so that the real-time captured frame is continuous.

$\bullet$ Its design leverages feasible low-cost devices and achieves reliable performance in real-time applications.

In the rest of this paper, Section II reviews the literature solutions related to our work. Next, Section III describes the overview system structure, includes system platform introduction, signal processing chains as well as surrounding reflection signals removing. Then, Section IV proposes the technical details to combine FMCW radar with antenna array to collect 3D array which represents the power at specific spacial voxels. Section V presents a novel solution that uses deep learning to make our scheme recognize and remove bad reflection frames, followed by Section VI evaluates the performance and feasibility of the whole system.

II Related work

Related works in this field include traditional sensors solutions, computer vision solutions and RF solutions. Recently, all of those solutions have been investigated for many smart home applications.

II-A Sensors and Computer Vision solutions

Accelerometer and Gyroscope[4] are the most common sensors applied to collect human motion data. Besides that, Inertial measurement unit (IMU) sensor which combines accelerometer and gyroscope sensor is also widely used to wearable devices[5]. Those sensors can collect linear acceleration, rotation angle, angular velocity of targets, so that human who wears sensors can be collected motion data. Based on raw data collected by those sensors, researchers proposed various algorithms to recognize human activities[6, 7, 8, 9]. Zhang et al. designed physical features based on physical parameters of human motion, then find the most critical physical features for human activities, thus to improve the recognition accuracy[6, 7]. Years later, Ferhat et al. investigated k-Nearest Neighbor (k-NN), Support Vector Machines (SVM), Gaussian Mixture Models (GMM) classification techniques to process raw data and found the best scheme to recognize human activities[8]. Recently, researchers from the UK explored how to use deep, convolutional and recurrent models to detect human activities[9]. In the meantime, many algorithms have been proposed based on computer vision techniques[10, 11, 12, 13, 14]. Sung et al. used RGB-D images to detect and track human motions, those images with depth information were properly processed to achieve capture purpose[10, 13]. Jalal’s team proposed a solution, which uses translation and scaling invariant features with depth videos to recognize human logging activity[11]. More recently, Kinect was widely used to collect human motion data because of its abundant APIs; researchers trained mechine models to do image segmentation for Kinect real-time video and capture coarse human outlines and motions[12, 14].

II-B RF Solutions

Radar sensors and RF devices usually used for military or wireless communication purpose. However, it has been recently considered for smart home applications because of its data confidentiality, and the performance does not depend on lighting conditions. Recent researches either based on FMCW radar [15, 16, 17, 18] or used off-the-shelf devices [19, 20, 21]. A research group of Massachusetts Institute of Technology (MIT) Adib et al. designed MIMO antenna sensor with FMCW technique to detect human move [15] and even capture human figure through a wall [17, 16]. Off-the-shelf devices such as ultrasonic sensor or walabot [22] were also investigated feasibility to recognize human activities [19, 20, 21]. Avrahami et al. proposed a human activity recognition scheme based on 2D heat maps generated by walabot, while Zhu et al. [20] applied traditional signal processing algorithms to filter and cluster raw data thus recognize human actions. Both of them achieve higher than 80% accuracy in their research tasks.

III System Design Overview

To enable real-time human image capturing with ambient radar in smart home scenario, we propose a solution: Human Image Capturing based on FMCW Radar sensors (HICFR) system. HICFR scans 3D surroundings reflections with FMCW chirps and 2D antenna array. While FMCW chirps are used to compute direct distance from the detected object to receive antenna, and 2D antenna array is placed to identify spatial directions. It emits FMCW chirps to scan 3D volume of surroundings, then received signals are processed to remove environment fix reflections,. After that, we calculate the reflection powers of any scanned voxel and construct it to 3D images, then a Deep Neural Network (DNN) based filter algorithm is designed to address noncontinuous reflection frames problem, thus capture real-time human activities.

III-A Sensing Platform

Since HICFR requires to emit FMCW chirps and collect received signals by 2D antenna array, there is an off-the-shelf radar sensor called Walabot [22] meets our requirement. Walabot has compact size and low-cost feature with a board size of $72mm*140mm$ and the average power is lower than -41dbm/MHz. The frequency range of FMCW chirp emitted by Walabot is 3.3GHz-10GHz, which is good enough to detect direct distance within 10 meters range based on gradient of FMCW chirp. It also contains 18 pair of antenna, which are arranged to 2D antenna array.

Figure 1 shows internal antenna array of Walabot. Walabot emits FMCW chirps to scan $\phi$ in horizontal direction and $\theta$ in vertical direction. Then it communicates to HICFR with USB port to send raw signals for further processing. The scanned area of walabot can be present as figure2 below:

Where $\theta$ is Elevation angle to detect the height of human, and $\phi$ is Wide angle to capture the width of human. $R$ is FMCW signals travel distance from transmit antenna to humans head, and $R^{\prime}$ is hypotenuse of triangle whose angle is $\theta$ and hypotenuse $R$ rotate $\phi$ degree, the scan range is the sector which triangle passed. In our case, $\theta$ is from $-45\degree$ to $45\degree$ and $\phi$ is from $-90\degree$ to $90\degree$. The direct travel distance $R$ can be calculated by FMCW properly with formula (1) and figure 3 as below:

[TABLE]

While $\bigtriangleup t$ is signal travel time from transmit antenna to object and reflect back to receive antenna, and $\bigtriangleup f$ is frequency difference of transmit and receive signals. $df/dt$ is slope of transmit or echo frequency chirp and $c$ is speed of light. To simplify description, Equation (1) and formula 3 are not considering doppler frequency shift effect.

III-B System Flowgraph

The complete HICFR system contains 3 phases. 1) Data collection and Calibration, 2) Coarse Visualization and 3) Fine Visualization. As shown in figure 4, environment sensing period is running ahead of detecting. First, HICFR emits FMCW chirps and records static background reflections, when HICFR starts to do object detection task, its 2D antenna array collects raw signals. Second phase is designed to convert signals to images. Since HICFR scans 3D surroundings with parameters $R,\theta$ and $\phi$, the raw received signals then being processed to represent power of every spatial points with different $R,\theta$ and $\phi$ , namely voxel in scanned area. Then HICFR subtracts recorded background reflections power, and removes environment fix reflection to get pure power information of changed objects, which is a 3D matrix $M$ with dimension of $(sizeX,sizeY,sizeZ)$, where $sizeX,sizeY,sizeZ$ can be computed by equations (2).

[TABLE]

In the equations above, $range(\cdot)$ is detect range of parameters, while $res(\cdot)$ is designated parameters’ sampling interval. Next, we propose a novel solution to achieve coarse-to-fine visualization. Because human body acts as an uneven reflector rather than a scatterer, thus some signals reflect back directly to antenna array, while other signals are deflected from normal path or even away from receive antenna, in this case, constructed 3D images may contain some ambiguous results. The third phase addresses this issue by using Machine Learning algorithm. We collect dataset for regular reflection and deflected reflection images, then train a Deep Neural Network (DNN) which contains Convolutional Layer, Pooling layer and Linear layer to recognize them. The trained DNN is placed to main program loop thus eliminates ambiguous frames from real-time stream.

As shown above, the key challenges and main contributions of HICFR are 1) Compute reflection powers of every voxel with $R,\theta$ and $\phi$ based on the received complex signals of antenna array, 2) Construct 2D/3D images with known 3D power matrix, 3) Address ambiguous images issue which caused by signal deflection with Deep Neural Network and achieve real-time filtering scheme.

IV Calibration and Visualization

In this section, we dive in the technical detail of HICFR. Since walabot antenna array collects RF-signals, which is complex signals, they can be represented by amplitude and phase as follows:

[TABLE]

Where $s_{t}$ is signals received at $t$ moment. $A_{t}$ is amplitude of signal at time $t$, $r$ is travel distance of signal and $\lambda$ is signals’ wavelength. Since received phase has linear function with travel distance, so $2\pi\frac{r}{\lambda}t$ is the signal phase when it reach to receive antenna at moment $t$.

Revisit to equation 3, due to the receiver is an antenna array, so $s_{t}$ should have more complex format to specify the signal is received by which receive antenna, note as $s_{n,t}$, where $n$ is $n_{th}$ antenna number, and $s_{n,t}$ is signals received by antenna $n$ at moment $t$.

Another parameter needs to be clarified is $r$. Since human body is a surface rather than a point, it reflects signals from different directions to all antennas, the received signals at moment $t$ of one antenna contains more than one points’ reflection, thus $r$ varies from multiple reflect points. Figure 5(a) shows when antenna array scans human body, his left hand $p_{1}$ reflects to antenna $a_{1},a_{2},a_{3},a_{4}$ as blue dot line, his right hand $p_{2}$ reflects to antenna array as red dot line. Based on above description, equation 4 is designated as follow:

[TABLE]

Suppose $p_{k}$ is $k_{th}$ points on the detected object, then $K$ is number of points being scanned, $r_{n,k}$ is signal travel from $p_{k}$ to antenna $a_{n}$.

IV-A Compute Voxel Power

Power of Direction: Based on equation 3 and 4, the problem can be decleared as: known signals $s_{n,t}$ received by antenna $a_{n}$ at moment $t$, then compute reflection power of every scanned points. Because both angles and distance property can be reflected to phase of received signal. More specifically, the power of specific angle $\phi,\theta$ can be refered by antenna array property, while the power of specific distance $r$ can be calculated by FMCW feature. Revisit to figure 5(a) and change antenna array panel to a plane figure, antenna $a_{1},a_{2},a_{3},a_{4}$ receive reflection from $p_{k}$, the coming direction of beam is $\phi$ as shown in both figure 5(b) and figure 2. While $\alpha_{1},\alpha_{2},\alpha_{3},\alpha_{4}$ are angles between antenna to $p_{k}$, and $d$ is distance between two antennas. Thus power of direction $\phi$ can be presented as $P(\phi)$ in equation 5:

[TABLE]

Where $N$ is how many antennas in the dimension. Because $s_{n,t}$ travel different distance for each antenna, and the difference can be represented by $nd\sin\phi$ as depicting with light blue color. Thus their phase change of antenna $n$ is $2\pi\frac{nd\sin\phi}{\lambda}$, $\lambda$ is signal wavelength.

Power of Distance: The travel distance of signals also related to the direct distance from point $p_{k}$ to antenna $a_{n}$. Frequency Modulated Continuous Wave measures reflection depth by calculating frequecy shift between transmit and receive chirp. Equation 1 shows the FMCW feature. We define $v$ is slope of frequency chirp versus time, where $v$ is equal to $df/dt$ in figure 3. So the power of distance $r_{k}$ can be calculated by phase change of $s_{t,n}$ as shown below in equation 6:

[TABLE]

where $r_{k}$ is signal travel distance from point $k$. $T$ is the duration of each chirp. Because $f=vt$ and $r/c=t_{travel}$, we can easily get the phase change is $2\pi ft_{travel}$, thus power of $r_{k}$ is summation over duration $T$ and total antenna number $N$.

Power of Voxel: Since HICFR scans 3D surroundings, reconsider situation shown at figure 5(b), where $p_{k}$ on same panel of antenna. However, points in 3D volume need three parameters to locate, either with ($r,\theta,\phi$) in spherical coordinate system or ($x,y,z$) in cartesian coordinate system. We choose spherical coordinate system because the power of $\theta$ and $\phi$ can be calculated based on our 2D antenna array. Figure 6 shows how it works:

The 2D antenna array is on $X-Y$ panel, where blocks is antenna. $d_{x}$ and $d_{y}$ are distance between two antenna in two dimensions. $Y-Z$ panel is the dimension drawn in figure 5(b), and $d_{y},\phi$ is $d,\phi$ in equation 5, while $\theta$ is elevation angle from $Y-Z$ panel to $R$. In 3D figure, $R$ is mapping to $Y-Z$ pannel as $YZ$ with $\cos\theta$, and it is mapping to $X-Z$ panel as $XZ$ with $\cos\phi$, where $\phi$ is wide angle from $YZ$ to $Z$ axis. Thus the distance change at $Y-Z$ panel for each antenna is $\cos\theta*nd_{y}*\sin\phi$ as blue line, that change at $X-Z$ panel is $\cos\phi*md_{x}*\sin\theta$ as light blue line shows.

[TABLE]

Since distance change represents phase change of signals, then we can calculate power of any voxel by equation 7. $s_{n,m,t}$ is the signal received by receive antenna $n$ from transmit antenna $m$ at time $t$.

IV-B Construct 3D Image

Remove Background Reflection: To get rid of environment reflections such as desks or walls, HICFR starts a sensing process before capture humans, name as calibration. Since background reflection is static and the reflection power is fixed, so that calibration sensing, calculating and recording the background reflection power of any voxel, after that, when HICFR starts human image capturing task, it subtracts the static background reflection power from the real-time reflection power. We need to make sure there is no human enters the lab during calibration period.

Construct 2D/3D Image: Once HICFR calculates the power of every voxel and removes background reflection power, it gets a 3D matrix $M$ with the dimension of ($sizeX,sizeY,sizeZ$), where $sizeX,sizeY,sizeZ$ can be refered from equation 2. Since a 2D image is related to either $(R,\theta)$, $(R,\phi)$ or even $(\theta,\phi)$. To make 2D image has a clear meaning, we choose to construct 2D image with distance and wide angle $(R,\phi)$. At first, we find the highest power from $M$, suppose the highest reflection power is from point at ($R_{a},\theta_{b},\phi_{c}$), then $M(R_{a},\theta_{b},\phi_{c})$ is the highest value in $M$, and $M(R,\theta_{b},\phi)$ is a 2D array because parameter $\theta$ is fixed as $\theta_{b}$. Thus we draw a 2D heatmap image based on $M(R,\theta_{b},\phi)$, where the color shows reflection power intensity, the darker color means the higher reflection power at $(R,\theta_{b},\phi)$. Figure 7(a) shows 2D image capturing scenario and its corresponding heatmap.

As can be seen from figure 7(a), the range of $\phi$ is from $-60\degree$ to $60\degree$, where $\phi$ is the angle from dash blue line to human, in this case, dash blue line is the base line in the middle of Walabot, thus $\phi$ is wide angle from base line to object. While 2D image only depicts the highest power layer of fixed $\theta$, 3D image shows more information about object width, height and location. Figure 7(b) shows how to construct 2D images to 3D images. HICFR uses marching cubes algorithm to draw vertices and faces of stacked images, then it uses nomarlized filter to remove low power points. It is very clear to see that human is at a shorter direct distance to radar, and the height of human is greater than the chairs in 3D vision.

V Filter Reflection

Another challenge of real-time human image capturing is signal deviation. Since human body is not a plane surface, especially when human moves, the surface of body is extremely deformed. As a result, while our antenna array transmits siganls and scans human body, only signals that close to normal surface are reflected back toward the antennas. Other signals may be deviated from another routes and back to receiver, which makes our antenna ”misunderstand” the real distance and angle from object. This scenario is shown in figure 9:

In this case, the distance between human chest and leg is not quite large, however, signals transmitted from antenna array travel to human leg and deviate it’s coming route, thus receive antenna gets signals from $r_{3}\theta_{3}\phi_{3}$, where $r_{3}=d_{1}+d_{2}$, so that antenna ”misunderstand” human leg position with wrong distance and angles, and it results in deformed 3D shape. To address this issue, we design a Deep Neural Network to recognize whether current 3D figure is deformed or not, and remove them from image capture stream.

DNN Recognization: We use transfer learning technique to solve this problem more efficiently. Due to it’s a image processing problem, the proper Deep Neural Network should have Convolutional layer to reduce possible parameters and amount of calculation. Based on that, we choose resNet18 to classify our 3D image. Our contribution is 1) Collect regular and ambiguous images used as training dataset, 2) Change the network structure of resNet18 to make the DNN convergence faster, 3) Real-time load trained DNN parameters and handle recognization task in mainstream.

We collect training dataset from real human activities, while one person walks around in the lab, we construct 3D images and concatenate them as 3D videos. Then we classify them manually into 2 categories: regular frames and ambiguous frames. Figure 8 shows samples of dataset, while figure 8(a) shows regular 3D reflection power and 8(b) has ambiguous images. As can be seen from the dataset, the regular frames show human 3D position very clear, and the ambiguous frames always ”misunderstand” location of some part of human body.

Change resNet18 Structure: The first step to apply transfer learning is changing the last Fully Connect(FC) layer, the last FC layer dimension of normal resNet18 is $(512,1000)$, which means $in\ feature$ to FC layer is $512$ and output $1000$ features. The $1000$ $out\ feature$ usually feed into $softmax$ functions to be classified into $1000$ categories. In our design, we only have $2$ categories: regular and ambiguous. Then we change the dimension of last FC layer to be $(512,2)$. The second change of original resNet18 is changing the pooling layer before FC layer. Resnet18 uses Average Pooling layer to compress features to $512$, but Average Pooling sometimes cannot extract good features because it takes all into count and results an average value. Since our dataset images have strong edges, and Max Pooling extracts the most important or extreme features. So we change the pooling layer to the same size of Max Pooling layer and compare the different convergence of them.

Train DNN: The DNN is trained with mini-batch strategy to make it converge more smoothly. We use $CrossEntropyLoss$ as loss function shown in equation 8. Where $x$ is output of DNN, whose dimension is $(minibatchsize,2)$, and $label$ is labels for one minibatch data with dimension $(minibatchsize,1)$. We use SGD optimizer to update parameters with $learning\ rate=0.01$ and $momentum=0.9$, and a $lr\ schedular$ is applied to adjust learning rate with $stepsize=7$ and $gamma=0.1$. Then we compare running loss and accuracy of each iterations in figure 11. Note that running loss and accuracy will be cleared after one epoch.

[TABLE]

Figure 11(a) shows the original performance of resnet18 and figure 11(b) is our DNN result. It is clearly to see our DNN converges faster and has less strong vibration compare to original resnet18.

VI Performance

The whole process of HRCIF results in figure 10. The first row records real human motions, the second row is the results before filtering, and the third row is a final result of detecting human. At the very beginning, human is standing on the right of radar with a wide angle of $60\degree$, where the cube in row 2 and 3 stand around $\phi=60\degree$ and $R=110cm$. With human moves close to radar from frame $1-3$, our captured images show wide angle $\phi$ and direct distance $R$ are decreasing gradually. While human move away from radar, the wide angle $\phi$ and direct distance $R$ are increasing. During this time, the frame ahead of last frame is ”bad frame”, so our DNN detects and recognizes the ”misunderstanding”, thus hold previous frame to the current one. More experiments are designed to see if captured stream can be used to recognize human activities such as walk, run, jump and fall. Our results show all activities stream captured by HICFR can be easily recognized by human with accuracy more than 90%.

VII Conclusion

In general, we propose a real-time 3D human image capturing scheme based on radar, this solution not only localizes human position precisely in stream, but also protect human’s privacy. Different human activities captured by our radar system results can be easily recognized by human eyes. Our future work will focus on designing a Recurrent Neural Network (RNN) to recognize human activities with our visualization result in real-time.