In-Home Daily-Life Captioning Using Radio Signals
Lijie Fan, Tianhong Li, Yuan Yuan, Dina Katabi

TL;DR
RF-Diary is a novel radio signal-based model that generates textual descriptions of daily life activities in homes, preserving privacy and functioning effectively in dark or occluded environments, unlike traditional video captioning methods.
Contribution
The paper introduces RF-Diary, a new radio signal analysis model for privacy-preserving in-home activity captioning, leveraging multi-modal training and 3D dynamics understanding.
Findings
RF-Diary accurately captions daily activities in various conditions.
It outperforms video-based captioning in dark and occluded settings.
The model demonstrates robustness in real-world home environments.
Abstract
This paper aims to caption daily life --i.e., to create a textual description of people's activities and interactions with objects in their homes. Addressing this problem requires novel methods beyond traditional video captioning, as most people would have privacy concerns about deploying cameras throughout their homes. We introduce RF-Diary, a new model for captioning daily life by analyzing the privacy-preserving radio signal in the home with the home's floormap. RF-Diary can further observe and caption people's life through walls and occlusions and in dark settings. In designing RF-Diary, we exploit the ability of radio signals to capture people's 3D dynamics, and use the floormap to help the model learn people's interactions with objects. We also use a multi-modal feature alignment training scheme that leverages existing video-based captioning datasets to improve the performance of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Human Pose and Action Recognition · Video Surveillance and Tracking Methods
