LightLLM: A Versatile Large Language Model for Predictive Light Sensing

TL;DR

LightLLM is a novel approach that fine-tunes pre-trained large language models for light sensing tasks by integrating sensor data and environmental context, achieving superior accuracy with minimal retraining.

Contribution

It introduces a lightweight, adaptable framework for applying large language models to light-based sensing, combining sensor data encoding, contextual prompts, and fusion layers.

Findings

Achieves 4.4x better localization accuracy in unseen environments.

Outperforms state-of-the-art methods in indoor solar estimation.

Surpasses ChatGPT-4 in light sensing tasks with direct prompting.

Abstract

We propose LightLLM, a model that fine tunes pre-trained large language models (LLMs) for light-based sensing tasks. It integrates a sensor data encoder to extract key features, a contextual prompt to provide environmental information, and a fusion layer to combine these inputs into a unified representation. This combined input is then processed by the pre-trained LLM, which remains frozen while being fine-tuned through the addition of lightweight, trainable components, allowing the model to adapt to new tasks without altering its original parameters. This approach enables flexible adaptation of LLM to specialized light sensing tasks with minimal computational overhead and retraining effort. We have implemented LightLLM for three light sensing tasks: light-based localization, outdoor solar forecasting, and indoor solar estimation. Using real-world experimental datasets, we demonstrate that LightLLM significantly outperforms state-of-the-art methods, achieving 4.4x improvement in localization accuracy and 3.4x improvement in indoor solar estimation when tested in previously unseen environments. We further demonstrate that LightLLM outperforms ChatGPT-4 with direct prompting, highlighting the advantages of LightLLM's specialized architecture for sensor data fusion with textual prompts.