Model-Attentive Ensemble Learning for Sequence Modeling

TL;DR

This paper introduces MAES, a novel ensemble learning approach with attention-based gating that adapts to temporal shifts in medical time-series data, significantly improving prediction accuracy.

Contribution

The paper proposes MAES, a mixture of experts model with attention gating, specifically designed to handle temporal conditional shifts in sequence modeling.

Findings

MAES outperforms existing sequence models on datasets with temporal shift.

Attention-based gating effectively specializes experts for different sequence dynamics.

MAES demonstrates significant improvement in medical time-series prediction accuracy.

Abstract

Medical time-series datasets have unique characteristics that make prediction tasks challenging. Most notably, patient trajectories often contain longitudinal variations in their input-output relationships, generally referred to as temporal conditional shift. Designing sequence models capable of adapting to such time-varying distributions remains a prevailing problem. To address this we present Model-Attentive Ensemble learning for Sequence modeling (MAES). MAES is a mixture of time-series experts which leverages an attention-based gating mechanism to specialize the experts on different sequence dynamics and adaptively weight their predictions. We demonstrate that MAES significantly out-performs popular sequence models on datasets subject to temporal shift.