IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor data

TL;DR

This paper introduces IoT-LLM, a framework that enhances large language models' reasoning abilities with real-world sensor data, improving their performance on IoT-related tasks through data preprocessing, knowledge retrieval, and chain-of-thought prompting.

Contribution

The paper proposes a novel IoT-LLM framework that systematically integrates IoT sensor data with LLMs, including a new benchmark for IoT-sensory reasoning tasks.

Findings

IoT-LLM significantly improves reasoning performance on IoT tasks.

Models like GPT-4o-mini achieve 49.4% average improvement.

The framework effectively combines perception augmentation with LLMs.

Abstract

Large Language Models excel in textual tasks but often struggle with physical-world reasoning tasks. Inspired by human cognition, where perception is fundamental to reasoning, we explore augmenting LLMs with enhanced perception abilities using Internet of Things (IoT) data and pertinent knowledge. In this work, we systematically study LLMs' capability to address IoT-sensory tasks by augmenting their perception and knowledge base, and then propose a unified framework, IoT-LLM, to enhance such capability. In IoT-LLM, we customize three steps: preprocessing IoT data into suitable formats, expanding LLMs knowledge via IoT-oriented retrieval-augmented generation and activating LLMs commonsense knowledge through chain-of-thought prompting. We design a benchmark comprising five real-world tasks with varying data types and reasoning complexities to evaluate the performance of IoT-LLM. Experimental results reveal that IoT-LLM significantly improves the performance of IoT-sensory task reasoning of LLMs, with models like GPT-4o-mini showing a 49.4% average improvement over previous methods.