Project Hermes: A Model-Agnostic Validation Layer for Wearable Health Prediction Systems

TL;DR

Project Hermes introduces a model-agnostic validation layer for wearable health prediction systems that uses LLMs and Bayesian updates to reduce false alarms and improve trustworthiness in real-world deployment.

Contribution

It presents a novel validation framework that operates downstream of existing predictors, using LLMs for user-specific validation without making diagnoses or predictions.

Findings

34% reduction in false positive rate

Maintained 89% sensitivity in migraine prediction

Achieved mean lead time of 4.2 hours before symptoms

Abstract

The deployment of wearable-based health prediction systems has accelerated rapidly, yet these systems face a fundamental challenge: they generate alerts under substantial uncertainty without principled mechanisms for user-specific validation. While large language models (LLMs) have been increasingly applied to healthcare tasks, existing work focuses predominantly on diagnosis generation and risk prediction rather than post-prediction validation of detected signals. We introduce Project Hermes, a model-agnostic validation layer that treats signal confirmation as a sequential decision problem. Hermes operates downstream of arbitrary upstream predictors, using LLM-generated contextual queries to elicit targeted user feedback and performing Bayesian confidence updates to distinguish true positives from false alarms. In a 60-day longitudinal case study of migraine prediction, Hermes achieved a 34% reduction in false positive rate (from 61.7% to 12.5%) while maintaining 89% sensitivity, with mean lead time of 4.2 hours before symptom onset. Critically, Hermes does not perform diagnosis or make novel predictions; it validates whether signals detected by upstream models are clinically meaningful for specific individuals at specific times. This work establishes validation as a first-class computational problem distinct from prediction, with implications for trustworthy deployment of consumer health AI systems.