RFMask: A Simple Baseline for Human Silhouette Segmentation with Radio Signals

TL;DR

RFMask introduces a radio signal-based method for human silhouette segmentation that overcomes optical camera limitations like low light and occlusion, achieving promising results in challenging scenarios.

Contribution

This work presents the first framework for human silhouette segmentation using millimeter wave radio signals, providing a new baseline for vision tasks with radio data.

Findings

Achieved effective silhouette segmentation in low-light and occlusion scenarios.

Collected a large dataset with radio and camera frames for human activity analysis.

Demonstrated superior performance over traditional camera-based methods in challenging conditions.

Abstract

Human silhouette segmentation, which is originally defined in computer vision, has achieved promising results for understanding human activities. However, the physical limitation makes existing systems based on optical cameras suffer from severe performance degradation under low illumination, smoke, and/or opaque obstruction conditions. To overcome such limitations, in this paper, we propose to utilize the radio signals, which can traverse obstacles and are unaffected by the lighting conditions to achieve silhouette segmentation. The proposed RFMask framework is composed of three modules. It first transforms RF signals captured by millimeter wave radar on two planes into spatial domain and suppress interference with the signal processing module. Then, it locates human reflections on RF frames and extract features from surrounding signals with human detection module. Finally, the extracted features from RF frames are aggregated with an attention based mask generation module. To verify our proposed framework, we collect a dataset containing 804,760 radio frames and 402,380 camera frames with human activities under various scenes. Experimental results show that the proposed framework can achieve impressive human silhouette segmentation even under the challenging scenarios(such as low light and occlusion scenarios) where traditional optical-camera-based methods fail. To the best of our knowledge, this is the first investigation towards segmenting human silhouette based on millimeter wave signals. We hope that our work can serve as a baseline and inspire further research that perform vision tasks with radio signals. The dataset and codes will be made in public.