Long-range Modeling and Processing of Multimodal Event Sequences

TL;DR

This paper introduces a novel multimodal temporal point process framework that extends language models to incorporate visual data, enabling long-range understanding and generation of rich, multimodal event sequences.

Contribution

It presents a new approach combining sequence compression and a two-stage training paradigm to handle long multimodal sequences effectively.

Findings

Outperforms state-of-the-art in predictive accuracy

Generates high-quality textual analyses of event sequences

Effectively models long-range dependencies in multimodal data

Abstract

Temporal point processes (TPPs) have emerged as powerful tools for modeling asynchronous event sequences. While recent advances have extended TPPs to handle textual information, existing approaches are limited in their ability to generate rich, multimodal content and reason about event dynamics. A key challenge is that incorporating multimodal data dramatically increases sequence length, hindering the ability of attention-based models to generate coherent, long-form textual descriptions that require long-range understanding. In this paper, we propose a novel framework that extends LLM-based TPPs to the visual modality, positioning text generation as a core capability alongside time and type prediction. Our approach addresses the long-context problem through an adaptive sequence compression mechanism based on temporal similarity, which reduces sequence length while preserving essential patterns. We employ a two-stage paradigm of pre-training on compressed sequences followed by supervised fine-tuning for downstream tasks. Extensive experiments, including on the challenging DanmakuTPP-QA benchmark, demonstrate that our method outperforms state-of-the-art baselines in both predictive accuracy and the quality of its generated textual analyses.