Features Fusion Framework for Multimodal Irregular Time-series Events

TL;DR

This paper introduces a novel features fusion framework based on LSTM for modeling multimodal irregular time-series events, effectively capturing complex nonlinear relationships and temporal dependencies, and outperforming existing methods.

Contribution

The proposed framework uniquely combines feature extraction, nonlinear correlation modeling, and feature gating to handle irregular, multimodal time-series data with complex relationships.

Findings

Significantly outperforms existing methods in AUC and AP metrics.

Effectively captures complex nonlinear relationships and temporal dependencies.

Demonstrated on MIMIC-III dataset with superior results.

Abstract

Some data from multiple sources can be modeled as multimodal time-series events which have different sampling frequencies, data compositions, temporal relations and characteristics. Different types of events have complex nonlinear relationships, and the time of each event is irregular. Neither the classical Recurrent Neural Network (RNN) model nor the current state-of-the-art Transformer model can deal with these features well. In this paper, a features fusion framework for multimodal irregular time-series events is proposed based on the Long Short-Term Memory networks (LSTM). Firstly, the complex features are extracted according to the irregular patterns of different events. Secondly, the nonlinear correlation and complex temporal dependencies relationship between complex features are captured and fused into a tensor. Finally, a feature gate are used to control the access frequency of different tensors. Extensive experiments on MIMIC-III dataset demonstrate that the proposed framework significantly outperforms to the existing methods in terms of AUC (the area under Receiver Operating Characteristic curve) and AP (Average Precision).