HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokens

TL;DR

This paper introduces history tokens to enhance transformer models for event sequence classification, addressing their inability to effectively summarize sequence history and improve local context understanding, leading to better performance across various domains.

Contribution

The paper proposes history tokens, a novel method for accumulating sequence history in transformers, improving classification accuracy in event sequence tasks.

Findings

Significant performance improvements in finance, e-commerce, and healthcare tasks.

History tokens effectively capture sequence history and local context.

Transformers with history tokens outperform traditional models in sequence classification.

Abstract

Deep learning has achieved remarkable success in modeling sequential data, including event sequences, temporal point processes, and irregular time series. Recently, transformers have largely replaced recurrent networks in these tasks. However, transformers often underperform RNNs in classification tasks where the objective is to predict future targets. The reason behind this performance gap remains largely unexplored. In this paper, we identify a key limitation of transformers: the absence of a single state vector that provides a compact and effective representation of the entire sequence. Additionally, we show that contrastive pretraining of embedding vectors fails to capture local context, which is crucial for accurate prediction. To address these challenges, we introduce history tokens, a novel concept that facilitates the accumulation of historical information during next-token prediction pretraining. Our approach significantly improves transformer-based models, achieving impressive results in finance, e-commerce, and healthcare tasks. The code is publicly available on GitHub.