METHOD: Modular Efficient Transformer for Health Outcome Discovery

TL;DR

This paper introduces \\METHOD, a novel transformer architecture tailored for healthcare data that improves prediction accuracy and efficiency in clinical sequence modelling, addressing unique challenges of electronic health records.

Contribution

THOD combines patient-aware attention, adaptive sliding window schemes, and a U-Net inspired architecture to enhance long sequence processing in healthcare applications.

Findings

Outperforms state-of-the-art models on MIMIC-IV data.

Maintains stable performance across different sequence lengths.

Better preserves clinical hierarchies in learned embeddings.

Abstract

Recent advances in transformer architectures have revolutionised natural language processing, but their application to healthcare domains presents unique challenges. Patient timelines are characterised by irregular sampling, variable temporal dependencies, and complex contextual relationships that differ substantially from traditional language tasks. This paper introduces \METHOD~(Modular Efficient Transformer for Health Outcome Discovery), a novel transformer architecture specifically designed to address the challenges of clinical sequence modelling in electronic health records. \METHOD~integrates three key innovations: (1) a patient-aware attention mechanism that prevents information leakage whilst enabling efficient batch processing; (2) an adaptive sliding window attention scheme that captures multi-scale temporal dependencies; and (3) a U-Net inspired architecture with dynamic skip connections for effective long sequence processing. Evaluations on the MIMIC-IV database demonstrate that \METHOD~consistently outperforms the state-of-the-art \ETHOS~model, particularly in predicting high-severity cases that require urgent clinical intervention. \METHOD~exhibits stable performance across varying inference lengths, a crucial feature for clinical deployment where patient histories vary significantly in length. Analysis of learned embeddings reveals that \METHOD~better preserves clinical hierarchies and relationships between medical concepts. These results suggest that \METHOD~represents a significant advancement in transformer architectures optimised for healthcare applications, providing more accurate and clinically relevant predictions whilst maintaining computational efficiency.