IoT-LM: Large Multisensory Language Models for the Internet of Things

TL;DR

This paper introduces IoT-LM, a large multisensory language model designed for IoT data, leveraging a vast dataset and a new adapter layer to enhance IoT understanding, reasoning, and interaction capabilities.

Contribution

The paper presents IoT-LM, the first multisensory language model for IoT, supported by MultiIoT dataset and a novel adapter layer for multisensory pre-training and instruction tuning.

Findings

Significant improvements on 8 IoT classification tasks

Enhanced interactive question-answering and reasoning with IoT data

Open-source release of data and modeling framework

Abstract

The Internet of Things (IoT) network integrating billions of smart physical devices embedded with sensors, software, and communication technologies is a critical and rapidly expanding component of our modern world. The IoT ecosystem provides a rich source of real-world modalities such as motion, thermal, geolocation, imaging, depth, sensors, and audio to recognize the states of humans and physical objects. Machine learning presents a rich opportunity to automatically process IoT data at scale, enabling efficient inference for understanding human wellbeing, controlling physical devices, and interconnecting smart cities. To realize this potential, we introduce IoT-LM, an open-source large multisensory language model tailored for the IoT ecosystem. IoT-LM is enabled by two technical contributions: the first is MultiIoT, the most expansive unified IoT dataset to date, encompassing over 1.15 million samples from 12 modalities and 8 tasks prepared for multisensory pre-training and instruction-tuning. The second is a new multisensory multitask adapter layer to condition pre-trained large language models on multisensory IoT data. Not only does IoT-LM yield substantial improvements on 8 supervised IoT classification tasks, but it also demonstrates new interactive question-answering, reasoning, and dialog capabilities conditioned on IoT sensors. We release IoT-LM's data sources and new multisensory language modeling framework.