Large Model for Small Data: Foundation Model for Cross-Modal RF Human Activity Recognition

TL;DR

This paper introduces FM-Fi, a cross-modal framework that leverages vision-based foundation models to improve RF-based human activity recognition, especially in small data scenarios, through knowledge distillation and few-shot learning.

Contribution

The paper presents a novel cross-modal contrastive knowledge distillation method and a framework that adapts vision-based foundation models for RF HAR, enabling zero-shot and few-shot learning.

Findings

FM-Fi achieves comparable performance to vision-based methods.

It demonstrates strong generalizability across different environments.

The framework effectively enhances RF HAR with limited data.

Abstract

Radio-Frequency (RF)-based Human Activity Recognition (HAR) rises as a promising solution for applications unamenable to techniques requiring computer visions. However, the scarcity of labeled RF data due to their non-interpretable nature poses a significant obstacle. Thanks to the recent breakthrough of foundation models (FMs), extracting deep semantic insights from unlabeled visual data become viable, yet these vision-based FMs fall short when applied to small RF datasets. To bridge this gap, we introduce FM-Fi, an innovative cross-modal framework engineered to translate the knowledge of vision-based FMs for enhancing RF-based HAR systems. FM-Fi involves a novel cross-modal contrastive knowledge distillation mechanism, enabling an RF encoder to inherit the interpretative power of FMs for achieving zero-shot learning. It also employs the intrinsic capabilities of FM and RF to remove extraneous features for better alignment between the two modalities. The framework is further refined through metric-based few-shot learning techniques, aiming to boost the performance for predefined HAR tasks. Comprehensive evaluations evidently indicate that FM-Fi rivals the effectiveness of vision-based methodologies, and the evaluation results provide empirical validation of FM-Fi's generalizability across various environments.