Looking around you: external information enhances representations for event sequences

TL;DR

This paper introduces a method to incorporate external information from co-occurring event sequences to enhance representation learning, significantly improving model performance across diverse domains like finance and e-commerce.

Contribution

It proposes a novel aggregation approach using attention mechanisms to leverage multiple user sequences, outperforming traditional isolated sequence processing.

Findings

Kernel attention improves ROC-AUC scores across datasets.

Mean pooling provides a smaller but significant performance boost.

Method supports efficient fine-tuning on existing encoders.

Abstract

Representation learning produces models in different domains, such as store purchases, client transactions, and general people's behaviour. However, such models for event sequences usually process each sequence in isolation, ignoring context from ones that co-occur in time. This limitation is particularly problematic in domains with fast-evolving conditions, like finance and e-commerce, or when certain sequences lack recent events. We develop a method that aggregates information from multiple user representations, augmenting a specific user for a scenario of multiple co-occurring event sequences, achieving better quality than processing each sequence independently. Our study considers diverse aggregation approaches, ranging from simple pooling techniques to trainable attention-based Kernel attention aggregation, that can highlight more complex information flow from other users. The proposed methods operate on top of an existing encoder and support its efficient fine-tuning. Across six diverse event sequence datasets (finance, e-commerce, education, etc.) and downstream tasks, Kernel attention improves ROC-AUC scores, both with and without fine-tuning, while mean pooling yields a smaller but still significant gain.