Byte-token Enhanced Language Models for Temporal Point Processes Analysis

TL;DR

This paper introduces Language-TPP, a novel framework combining Temporal Point Processes with Large Language Models using byte-token temporal encoding, achieving state-of-the-art results in Web event sequence modeling and enabling new applications.

Contribution

The paper proposes a new temporal encoding mechanism with byte-tokens that allows seamless integration of TPPs and LLMs without TPP-specific modifications, improving Web event analysis.

Findings

Achieved state-of-the-art performance on TPP benchmarks.

Enhanced event description quality and sentiment alignment.

Effective modeling of temporal and textual patterns in Web data.

Abstract

Temporal Point Processes (TPPs) have been widely used for modeling event sequences on the Web, such as user reviews, social media posts, and online transactions. However, traditional TPP models often struggle to effectively incorporate the rich textual descriptions that accompany these events, while Large Language Models (LLMs), despite their remarkable text processing capabilities, lack mechanisms for handling the temporal dynamics inherent in Web-based event sequences. To bridge this gap, we introduce Language-TPP, a unified framework that seamlessly integrates TPPs with LLMs for enhanced Web event sequence modeling. Our key innovation is a novel temporal encoding mechanism that converts continuous time intervals into specialized byte-tokens, enabling direct integration with standard language model architectures for TPP modeling without requiring TPP-specific modifications. This approach allows Language-TPP to achieve state-of-the-art performance across multiple TPP benchmarks, including event time prediction and type prediction, on real-world Web datasets spanning e-commerce reviews, social media and online Q&A platforms. More importantly, we demonstrate that our unified framework unlocks new capabilities for TPP research: incorporating temporal information improves the quality of generated event descriptions, as evidenced by enhanced ROUGE-L scores, and better aligned sentiment distributions. Through comprehensive experiments, including qualitative analysis of learned distributions and scalability evaluations on long sequences, we show that Language-TPP effectively captures both temporal dynamics and textual patterns in Web user behavior, with important implications for content generation, user behavior understanding, and Web platform applications. Code is available at https://github.com/qykong/Language-TPP.