On Non-asymptotic Theory of Recurrent Neural Networks in Temporal Point Processes

TL;DR

This paper provides a non-asymptotic theoretical analysis of recurrent neural network-based temporal point processes, establishing bounds on their generalization errors and demonstrating their effectiveness with shallow architectures.

Contribution

It introduces the first non-asymptotic excess risk bounds for RNN-TPPs, showing that shallow networks can achieve vanishing errors and bridging theory with practical applications.

Findings

RNN-TPPs with up to four layers can achieve vanishing generalization errors.

Characterization of the complexity of multi-layer RNN classes.

Construction of $ anh$ neural networks for approximating dynamic event intensities.

Abstract

Temporal point process (TPP) is an important tool for modeling and predicting irregularly timed events across various domains. Recently, the recurrent neural network (RNN)-based TPPs have shown practical advantages over traditional parametric TPP models. However, in the current literature, it remains nascent in understanding neural TPPs from theoretical viewpoints. In this paper, we establish the excess risk bounds of RNN-TPPs under many well-known TPP settings. We especially show that an RNN-TPP with no more than four layers can achieve vanishing generalization errors. Our technical contributions include the characterization of the complexity of the multi-layer RNN class, the construction of $\tanh$ neural networks for approximating dynamic event intensity functions, and the truncation technique for alleviating the issue of unbounded event sequences. Our results bridge the gap between TPP's application and neural network theory.