RF-Behavior: A Multimodal Radio-Frequency Dataset for Human Behavior and Emotion Analysis

TL;DR

RF-Behavior is a comprehensive multimodal radio frequency dataset that combines radar, RFID, inertial, and visual data to enable advanced human behavior and emotion analysis, filling a critical gap in existing datasets.

Contribution

The paper introduces RF-Behavior, a novel multimodal RF dataset with synchronized sensors capturing gestures, activities, and emotions, supporting multi-task learning research.

Findings

Sensor placement affects modality complementarity.

Distinct performance across behavioral categories.

Multimodal data enhances behavior recognition.

Abstract

Recent research has demonstrated the complementary nature of camera-based and inertial data for modeling human gestures, activities, and sentiment. Yet, despite its growing importance for environmental sensing as well as the advance of joint communication and sensing for prospective WiFi and 6G standards, a dataset that integrates these modalities with radio frequency data (radar and RFID) remains rare. We introduce RF-Behavior, a multimodal radio frequency dataset for comprehensive human behavior and emotion analysis. We collected data from 44 participants performing 21 gestures, 10 activities, and 6 sentiment expressions. Data were captured using synchronized sensors, including 13 radars (8 ground-mounted and 5 ceiling-mounted), 6 to 8 RFID tags (attached to each arm) and LoRa. Inertial measurement units (IMUs) and 24 infrared cameras are used to provide precise motion ground truth. RF-Behavior provides a unified multimodal dataset spanning the full spectrum of human behavior -- from brief gestures to activities and emotional states -- enabling research on multi-task learning across motion and emotion recognition. Benchmark results demonstrate that the strategic sensor placement is complementary across modalities, with distinct performance characteristics across different behavioral categories.