The Law of Large Numbers and CLT for Non-stationary Markov Jump Processes Exhibiting Time-of-Day Effects

TL;DR

This paper establishes laws of large numbers and central limit theorems for non-stationary Markov jump processes with time-of-day effects, applicable to service and manufacturing systems with complex reward structures.

Contribution

It introduces new LLN and CLT results for non-stationary Markov jump processes with time-dependent transition rates and rewards, including periodic and resetting cases.

Findings

CLT approximations are accurate for various non-stationary models.

Theorems accommodate non-stationary rewards at jump and scheduled times.

Simulation confirms the effectiveness of the CLT in practical scenarios.

Abstract

In this paper, we develop a general law of large numbers and central limit theorem for cumulative reward processes associated with finite state Markov jump processes with non-stationary transition rates. Such models commonly arise in service operations and manufacturing applications in which time-of-day, day-of-week, and secular effects are of first-order importance in predicting system behavior. Our theorems allow for non-stationary reward environments that continuously accumulate reward, while also including contributions from non-stationary lump-sum rewards of random size that are collected at either jump times of the underlying process, jump times of a Poisson process modulated by the underlying process, or scheduled deterministic times. As part of our development, we also obtain a new central limit theorem for the special case in which the jump process transition rates and reward structure are periodic (as may occur over a weekly time interval), as well as for jump process models with resetting. We include a simulation study illustrating the quality of our CLT approximations for several non-stationary stochastic models.