Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition

TL;DR

This paper explores transforming eye-tracking sensor data into visualization images to improve human activity recognition with multimodal LLMs, offering a scalable and token-efficient approach.

Contribution

It introduces a visual prompting strategy converting eye-tracking data into images for MLLMs, systematically evaluated across datasets and visualization types.

Findings

Visual prompting enhances token efficiency for eye-tracking data.

Different visualization types impact recognition performance.

The approach enables MLLMs to reason over high-frequency sensor signals.

Abstract

Large Language Models (LLMs) have emerged as foundation models for IoT applications such as human activity recognition (HAR). However, directly applying high-frequency and multi-dimensional sensor data, such as eye-tracking data, leads to information loss and high token costs. To mitigate this, we investigate a visual prompting strategy that transforms sensor signals into data visualization images as an input to multimodal LLMs (MLLMs) using eye-tracking data. We conducted a systematic evaluation of MLLM-based HAR across three public eye-tracking datasets using three visualization types of timeline, heatmap, and scanpath, under varying temporal window sizes. Our findings suggest that visual prompting provides a token-efficient and scalable representation for eye-tracking data, highlighting its potential to enable MLLMs to effectively reason over high-frequency sensor signals in IoT contexts.