ARMA Processes with Discrete-Continuous Excitation: Compressibility Beyond Sparsity

TL;DR

This paper analyzes the information dimension rate and block information dimension of ARMA processes with discrete-continuous excitation, revealing that their compressibility matches that of their excitation noise, extending understanding beyond sparse sources.

Contribution

It establishes the relationship between IDR, BID, and $\\epsilon$-achievable compression rates for ARMA processes with mixed excitation noise, a novel extension beyond prior specific process types.

Findings

RID and $\\epsilon$-achievable rates equal to excitation noise

Samples can be compressed as much as sparse excitation noise

Results applicable to locally correlated data with finite or infinite memory

Abstract

R\'enyi Information Dimension (RID) plays a central role in quantifying the compressibility of random variables with singularities in their distribution, encompassing and extending beyond the class of sparse sources. The RID, from a high perspective, presents the average number of bits that is needed for coding the i.i.d. samples of a random variable with high precision. There are two main extensions of the RID for stochastic processes: information dimension rate (IDR) and block information dimension (BID). In addition, a more recent approach towards the compressibility of stochastic processes revolves around the concept of $\epsilon$-achievable compression rates, which treat a random process as the limiting point of finite-dimensional random vectors and apply the compressed sensing tools on these random variables. While there is limited knowledge about the interplay of the the BID, the IDR, and $\epsilon$-achievable compression rates, the value of IDR and BID themselves are known only for very specific types of processes, namely i.i.d. sequences (i.e., discrete-domain white noise) and moving-average (MA) processes. This paper investigates the IDR and BID of discrete-time Auto-Regressive Moving-Average (ARMA) processes in general, and their relations with $\epsilon$-achievable compression rates when the excitation noise has a discrete-continuous measure. To elaborate, this paper shows that the RID and $\epsilon$-achievable compression rates of this type of processes are equal to that of their excitation noise. In other words, the samples of such ARMA processes can be compressed as much as their sparse excitation noise, although the samples themselves are by no means sparse. The results of this paper can be used to evaluate the compressibility of various types of locally correlated data with finite- or infinite-memory as they are often modelled via ARMA processes.