Time-Dependent Representation for Neural Event Sequence Prediction

TL;DR

This paper introduces novel time-dependent event representations and regularization techniques for neural sequence prediction models, significantly improving accuracy on real-world, irregularly timed event data.

Contribution

It proposes new methods for incorporating time into neural sequence models and using event duration as regularization, addressing limitations of time-independent sequence modeling.

Findings

Proposed methods improve prediction accuracy across multiple datasets.

Time-aware representations outperform traditional time-agnostic models.

Regularization with next event duration enhances model training.

Abstract

Existing sequence prediction methods are mostly concerned with time-independent sequences, in which the actual time span between events is irrelevant and the distance between events is simply the difference between their order positions in the sequence. While this time-independent view of sequences is applicable for data such as natural languages, e.g., dealing with words in a sentence, it is inappropriate and inefficient for many real world events that are observed and collected at unequally spaced points of time as they naturally arise, e.g., when a person goes to a grocery store or makes a phone call. The time span between events can carry important information about the sequence dependence of human behaviors. In this work, we propose a set of methods for using time in sequence prediction. Because neural sequence models such as RNN are more amenable for handling token-like input, we propose two methods for time-dependent event representation, based on the intuition on how time is tokenized in everyday life and previous work on embedding contextualization. We also introduce two methods for using next event duration as regularization for training a sequence prediction model. We discuss these methods based on recurrent neural nets. We evaluate these methods as well as baseline models on five datasets that resemble a variety of sequence prediction tasks. The experiments revealed that the proposed methods offer accuracy gain over baseline models in a range of settings.