Temporal Tokenization Strategies for Event Sequence Modeling with Large Language Models

TL;DR

This paper systematically compares various temporal tokenization strategies for event sequence modeling with large language models, emphasizing the importance of aligning encoding methods with data distributions.

Contribution

It provides an empirical evaluation of different temporal encoding strategies, highlighting the significance of data-aware tokenization in LLM-based event modeling.

Findings

No single tokenization strategy is universally best.

Performance depends on matching tokenizer to data distribution.

Temporal tokenization is a crucial design choice in event modeling.

Abstract

Representing continuous time is a critical and under-explored challenge in modeling temporal event sequences with large language models (LLMs). Various strategies like byte-level representations or calendar tokens have been proposed. However, the optimal approach remains unclear, especially given the diverse statistical distributions of real-world event data, which range from smooth log-normal to discrete, spiky patterns. This paper presents a systematic empirical study of temporal tokenization for modeling event sequences with LLMs, comparing distinct encoding strategies: naive numeric strings, high-precision byte-level representations, human-semantic calendar tokens, classic uniform binning, and adaptive residual scalar quantization. We evaluate these strategies by fine-tuning LLMs on real-world datasets that exemplify these diverse distributions. Our analysis reveals that no single strategy is universally superior; instead, prediction performance depends heavily on aligning the tokenizer with the data's statistical properties, highlighting temporal tokenization as a critical yet often overlooked design dimension in LLM-based event modeling.