Benchmarking LLM Summaries of Multimodal Clinical Time Series for Remote Monitoring

TL;DR

This paper introduces an event-based evaluation framework for assessing the clinical accuracy of LLM-generated summaries of multimodal time-series data, revealing that traditional metrics often overlook critical event fidelity.

Contribution

It presents a novel event-aware evaluation protocol and benchmarks different summarization approaches, highlighting the importance of event-level correctness in clinical summaries.

Findings

Vision-based approach achieves 45.7% abnormality recall.

High semantic similarity does not guarantee event accuracy.

Event-aware metrics reveal gaps in existing summarization methods.

Abstract

Large language models (LLMs) can generate fluent clinical summaries of remote therapeutic monitoring time series. However, it remains unclear whether these narratives faithfully capture clinically significant events, such as sustained abnormalities. Existing evaluation metrics primarily focus on semantic similarity and linguistic quality, leaving event-level correctness largely unmeasured. To address this gap, we introduce an event-based evaluation framework for multimodal time-series summarization using the Technology-Integrated Health Management (TIHM)-1.5 dementia monitoring dataset. Clinically grounded daily events are derived through rule-based abnormal thresholds and temporal persistence criteria. Model-generated summaries are then aligned with these structured facts. Our evaluation protocol measures abnormality recall, duration recall, measurement coverage, and hallucinated event mentions. We benchmark three approaches: zero-shot prompting, statistical prompting, and a vision-based pipeline that uses rendered time-series visualizations. The results reveal a striking decoupling between conventional metrics and clinical event fidelity. Models that achieve high semantic similarity scores often exhibit near-zero abnormality recall. In contrast, the vision-based approach demonstrates the strongest event alignment, achieving 45.7% abnormality recall and 100% duration recall. These findings underscore the importance of event-aware evaluation to ensure reliable clinical time-series summarization.