A Multi-Agent LLM Network for Suggesting and Correcting Human Activity and Posture Annotations

TL;DR

This paper introduces a multi-agent LLM system that helps people accurately annotate their activities by combining self-reports with sensor data.

Contribution

The novel contribution is adapting the GLOSS system to improve activity recall and correct annotation errors using LLMs.

Findings

GLOSS achieves 63–75% agreement with human activity recall.

The system effectively identifies and corrects common human annotation errors.

Abstract

Accurate human activity recognition (HAR) is critical for health monitoring and behavior-aware systems. Developing reliable HAR models, however, requires large, high-quality labeled datasets that are challenging to collect in free-living settings. Although self-reports offer a practical solution for acquiring activity annotations, they are prone to recall biases, missing data, and human errors. Context-assisted recall can help participants remember their activities more accurately by providing visualizations of multiple data streams, but triangulating this information remains a burdensome and cognitively demanding task. In this work, we adapt GLOSS, a multi-agent LLM system that can triangulate self-reports and passive sensing data to assist participants in activity recall and annotation by suggesting the most likely activities. Our results show that GLOSS provides reasonable activity suggestions that align with human recall (63–75% agreement) and even effectively identifies and corrects common human annotation errors. These findings demonstrate the potential of LLM-powered, human-in-the-loop approaches to improve the quality and scalability of activity annotation in real-world HAR studies.

Figures