IMG2IMU: Translating Knowledge from Large-Scale Images to IMU Sensing Applications
Hyungjun Yoon, Hyeongheon Cha, Hoang C. Nguyen, Taesik Gong, Sung-Ju, Lee

TL;DR
This paper introduces IMG2IMU, a method that leverages large-scale image pre-training by converting IMU data into spectrograms, enabling improved performance on diverse IMU sensing tasks with limited data.
Contribution
The paper proposes a novel approach to transfer vision pre-training to IMU sensing by converting sensor data into spectrograms and introducing a sensor-aware contrastive pre-training method.
Findings
Outperforms sensor-only pre-trained baselines by 9.6% F1-score on average
Effective transfer of vision knowledge to IMU tasks with limited data
Demonstrates versatility across four different IMU sensing applications
Abstract
Pre-training representations acquired via self-supervised learning could achieve high accuracy on even tasks with small training data. Unlike in vision and natural language processing domains, pre-training for IMU-based applications is challenging, as there are few public datasets with sufficient size and diversity to learn generalizable representations. To overcome this problem, we propose IMG2IMU that adapts pre-trained representation from large-scale images to diverse IMU sensing tasks. We convert the sensor data into visually interpretable spectrograms for the model to utilize the knowledge gained from vision. We further present a sensor-aware pre-training method for images that enables models to acquire particularly impactful knowledge for IMU sensing applications. This involves using contrastive learning on our augmentation set customized for the properties of sensor data. Our…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Multimodal Machine Learning Applications · Advanced Image and Video Retrieval Techniques
MethodsContrastive Learning
