Vision-Aided Frame-Capture-Based CSI Recomposition for WiFi Sensing: A Multimodal Approach

TL;DR

This paper introduces a multimodal deep learning approach combining camera images and beamforming feedback matrices to improve the accuracy of WiFi channel state information recomposition, enhancing sensing capabilities.

Contribution

It proposes a novel vision-aided CSI recomposition method using multimodal deep learning to leverage spatial information from images alongside BFMs.

Findings

Multimodal approach outperforms single-modal methods in CSI recomposition accuracy.

Camera images compensate for spatial information gaps in BFMs.

Experimental validation on IEEE 802.11ac devices confirms improved performance.

Abstract

Recompositing channel state information (CSI) from the beamforming feedback matrix (BFM), which is a compressed version of CSI and can be captured because of its lack of encryption, is an alternative way of implementing firmware-agnostic WiFi sensing. In this study, we propose the use of camera images toward the accuracy enhancement of CSI recomposition from BFM. The key motivation for this vision-aided CSI recomposition is to draw a first-hand insight that the BFM does not fully involve spatial information to recomposite CSI and that this could be compensated by camera images. To leverage the camera images, we use multimodal deep learning, where the two modalities, i.e., images and BFMs, are integrated to recomposite the CSI. We conducted experiments using IEEE 802.11ac devices. The experimental results confirmed that the recomposition accuracy of the proposed multimodal framework is improved compared to the single-modal framework only using images or BFMs.