Foundation Models for Wearable Movement Data in Mental Health Research

TL;DR

This paper introduces PAT, a novel transformer-based foundation model for wearable movement data, achieving state-of-the-art mental health prediction performance and offering interpretability and efficiency for mental health research.

Contribution

The paper presents the first open source foundation model for wearable time-series data, adapting transformer architectures with novel techniques for mental health applications.

Findings

Achieves state-of-the-art performance in mental health prediction tasks

Lightweight and interpretable model suitable for clinical research

Pretrained on data from over 29,000 participants

Abstract

Pretrained foundation models and transformer architectures have driven the success of large language models (LLMs) and other modern AI breakthroughs. However, similar advancements in health data modeling remain limited due to the need for innovative adaptations. Wearable movement data offers a valuable avenue for exploration, as it's a core feature in nearly all commercial smartwatches, well established in clinical and mental health research, and the sequential nature of the data shares similarities to language. We introduce the Pretrained Actigraphy Transformer (PAT), the first open source foundation model designed for time-series wearable movement data. Leveraging transformer-based architectures and novel techniques, such as patch embeddings, and pretraining on data from 29,307 participants in a national U.S. sample, PAT achieves state-of-the-art performance in several mental health prediction tasks. PAT is also lightweight and easily interpretable, making it a robust tool for mental health research. GitHub: https://github.com/njacobsonlab/Pretrained-Actigraphy-Transformer/