A Study of the Allan Variance for Constant-Mean Non-Stationary Processes

TL;DR

This paper extends the Allan Variance to non-stationary processes, providing a theoretical framework that improves understanding and interpretation of AV in practical, real-world signals with non-stationary characteristics.

Contribution

It generalizes the Allan Variance to non-stationary processes, enabling more accurate analysis of signals with evolving covariance structures.

Findings

New theoretical form of AV for non-stationary processes

Simulation results demonstrate improved process differentiation

Enhanced interpretation of AV in applied signal analysis

Abstract

The Allan Variance (AV) is a widely used quantity in areas focusing on error measurement as well as in the general analysis of variance for autocorrelated processes in domains such as engineering and, more specifically, metrology. The form of this quantity is widely used to detect noise patterns and indications of stability within signals. However, the properties of this quantity are not known for commonly occurring processes whose covariance structure is non-stationary and, in these cases, an erroneous interpretation of the AV could lead to misleading conclusions. This paper generalizes the theoretical form of the AV to some non-stationary processes while at the same time being valid also for weakly stationary processes. Some simulation examples show how this new form can help to understand the processes for which the AV is able to distinguish these from the stationary cases and hence allow for a better interpretation of this quantity in applied cases.

Equations

$$\gamma (h)=\mathrm{cov}(X_{t+h},X_t),$$

$$\rho (h)=\frac{\gamma (h)}{\gamma (0)}.$$

$${\bar{X}}_t^{(n)}\equiv \frac{1}{n}\sum _{i=1}^n{X}_{t-n+i},$$

$${\mathrm{AVar}}_n\left(X_t\right)\equiv \frac{1}{2m^{\star }}\sum _{k=2n}^T\mathbb{E}\left[{\left({\bar{X}}_k^{(n)}-{\bar{X}}_{k-n}^{(n)}\right)}^2\right].$$

$${\widehat{\mathrm{AVar}}}_n\left(X_t\right)=\frac{1}{2m^{\star }}\sum _{k=2n}^T{\left({\bar{x}}_k^{(n)}-{\bar{x}}_{k-n}^{(n)}\right)}^2.$$

$${\mathrm{AVar}}_n\left(X_t\right)$$

$${\mathbf{X}}_t^{(n)}\equiv {\left[X_{t-n+1}\cdots X_t\right]}^T.$$

$${\mathbf{\Sigma }}_t^{(n)}\equiv \mathrm{var}\left({\mathbf{X}}_t^{(n)}\right),$$

$${\mathbf{\Gamma }}_t^{(n)}\equiv \mathrm{cov}\left({\mathbf{X}}_{t-n}^{(n)},{\mathbf{X}}_t^{(n)}\right),$$

$$\tilde{\gamma }(h)\equiv \frac{1}{2m^{\star }(n-h)}\sum _{t=2n}^T\sum _{s=0}^{n-h-1}\mathrm{cov}(X_{t-n-s-h},X_{t-n-s})$$

$$+\mathrm{cov}(X_{t-s-h},X_{t-s}).$$

$${\tilde{\gamma }}^{\ast }(h)\equiv \frac{1}{m^{\star }h}\sum _{t=2n}^T\sum _{s=1}^h\mathrm{cov}\left(X_{t-n+s-h},X_{t-n+s}\right).$$

$$\tilde{\gamma }(h)\equiv \frac{1}{m^{\star }(2n-h)}\sum _{t=2n}^T\sum _{s=0}^{2n-h-1}\mathrm{cov}\left(X_{t-s-h},X_{t-s}\right).$$

$${\mathrm{AVar}}_n=$$

$$X_t\sim \mathcal{N}(0,{\sigma }_t^2),$$

$$X_t=c_i\text{if}t\in b_i.$$

$$X_t^{(i)}=\varphi X_{t-1}^{(i)}+{ϵ}_t$$

$$X_t=X_t^{(i)}\text{if}t\in b_i.$$

$${\mathrm{AVar}}_n\left(X_t\right)\equiv \frac{1}{2m}\sum _{k=1}^m\mathbb{E}\left[{\left({\bar{X}}_{2k}^{(n)}-{\bar{X}}_{2k-1}^{(n)}\right)}^2\right],$$

$${\widetilde{\mathrm{AVar}}}_n\left(X_t\right)=\frac{1}{2m}\sum _{k=1}^m{\left({\bar{x}}_{2k}^{(n)}-{\bar{x}}_{2k-1}^{(n)}\right)}^2.$$

$${\mathbf{X}}_j^{(n)}\equiv {\left[X_{(j-1)n+1}\cdots X_{jn}\right]}^T.$$

$${\mathbf{\Sigma }}_{2k}^{(n)}\equiv \mathrm{var}\left({\mathbf{X}}_{2k}^{(n)}\right),{\mathbf{\Sigma }}_{2k-1}^{(n)}\equiv \mathrm{var}\left({\mathbf{X}}_{2k-1}^{(n)}\right)$$

$$\text{and}{\mathbf{\Gamma }}_k^{(n)}\equiv \mathrm{cov}\left({\mathbf{X}}_{2k-1}^{(n)},{\mathbf{X}}_{2k}^{(n)}\right).$$

$${\bar{\sigma }}_{2k}^{(n)}\equiv \frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n{\left({\mathbf{\Sigma }}_{2k}^{(n)}\right)}_{i,j},$$

$${\bar{\sigma }}_{2k-1}^{(n)}\equiv \frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n{\left({\mathbf{\Sigma }}_{2k-1}^{(n)}\right)}_{i,j},$$

$${\bar{\gamma }}_k^{(n)}\equiv \frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^n{\left({\mathbf{\Gamma }}_k^{(n)}\right)}_{i,j}.$$

$${\bar{\sigma }}^{(n)}\equiv \frac{1}{2m}\sum _{k=1}^m\left[{\bar{\sigma }}_{2k}^{(n)}+{\bar{\sigma }}_{2k-1}^{(n)}\right]\text{and}{\bar{\gamma }}^{(n)}\equiv \frac{1}{m}\sum _{k=1}^m{\bar{\gamma }}_k^{(n)}.$$

$$\tilde{\gamma }(h)\equiv \frac{1}{2m(n-h)}\sum _{k=1}^{2m}\sum _{s=1}^{n-h}\mathrm{cov}\left(X_{(k-1)n+s},X_{(k-1)n+s+h}\right).$$

$${\tilde{\gamma }}^{\ast }(h)\equiv \frac{1}{mh}\sum _{k=1}^m\sum _{s=1}^h\mathrm{cov}\left(X_{(2k-1)n+s-h},X_{(2k-1)n+s}\right).$$

$$\tilde{\gamma }(h)\equiv \frac{1}{m(2n-h)}\sum _{k=1}^m\sum _{s=1}^{2n-h}\mathrm{cov}\left(X_{(k-1)n+s},X_{(k-1)n+s+h}\right).$$

$${\mathrm{AVar}}_n$$

$$\mathbb{E}\left[{\left({\bar{X}}_{2k}^{(n)}-{\bar{X}}_{2k-1}^{(n)}\right)}^2\right]$$

Full text

I Introduction

The Allan Variance (AV) is a widely used quantity in areas going from engineering to physics where there is an interest in studying the stochastic stability of error measurements from various instruments such as, among others, clocks and oscillators. Its usefulness resides in the fact that it provides an extremely informative summary on the variance of time series or, more generally, of autocorrelated processes, especially when these are non-stationary and with infinite variance. Indeed, [8] underlined how the AV is a better measure of uncertainty compared to standard methods (e.g. moving average variance) for processes such as random walks and non-stationary Fractional Autoregressive-Moving Average (ARFIMA) models, while being considerably useful also for stationary processes. For these processes the AV has a well known form which can help detect the kind of process, for example, from the log-log plot of the AV of an observed signal. The behaviour and forms of the AV for stationary and some non-stationary processes was studied in [8] where the AV is used to detect and understand the process underlying a signal issued from different voltage measurements. However, there are many other applications for which the AV is of interest such as the detection of noise terms characterising inertial sensors [see 2] and many others [see 5, for an overview].

However, although the AV is extremely useful in the above settings, it is not known how it behaves when in the presence of other types of processes and whether it is able to distinguish between them. In this paper we intend to investigate the form of the AV for a particular class of processes which includes all those processes that have a constant mean but have a time-varying variance-covariance structure such as, for example, the Generalized AutoRegressive Conditional Heteroscedasticity (GARCH) models [see 1], while processes with specific forms of mean or higher-order non-stationarity are not considered since they can either be dealt with through statistical regression techniques or simply cannot be detected by the AV. In particular, we focus on those processes which are characterized by a dependence structure by blocks since they are common in settings such as longitudinal studies or sensor calibration for navigation engineering. In the latter cases, the AV is often approximated by that of other known stationary processes such as, for example, the bias-instability process whose AV is often approximated by that of a first-order autoregressive process [see, for example, 7]. Moreover, it is not clear whether the AV can actually help to distinguish between these processes and those processes for which its form is currently known. The latter aspect is of particular relevance since it could lead to an erroneous interpretation of the observed process, for example assuming stationarity when this is not the case and reaching false conclusions.

In order to deal with the above mentioned non-stationary processes, this paper intends to study the theoretical form of the AV when the covariance structure is non-stationary. The consequent advantage of this study is that, by considering the varying covariance structure in the AV definition, it extends the applicability of those approaches that make use of the AV and raises awareness on its limitations, and inappropriate interpretation, in distinguishing and identifying these processes from stationary ones. With this in mind, Section II briefly defines the AV and describes its theoretical form for those processes which have been considered up to now. Section III introduces the new theoretical form of both the overlapping and non-overlapping AV for processes whose covariance structure is non-stationary and shows how the form of the AV for stationary processes is a special case of this new form. In the same section, three case studies are presented which highlight the importance of these findings in order to better interpret processes through the AV. Finally Section IV concludes.

II Overview of the Allan Variance

To introduce the AV, let us first define $(X_{t})_{t=1,\ldots,T}$ as a weakly stationary, discrete time and regularly spaced stochastic process with a constant mean $\mu$ (i.e. $\mathbb{E}[X_{t}]=\mu$) and an autocovariance function defined as follows

[TABLE]

which depends solely on $h$, the distance between observations, with $\sigma_{X}^{2}=\gamma(0)$ being the process variance. We can consequently define the autocorrelation function as

[TABLE]

We consider the AV computed at dyadic scales ($\tau$) starting from local averages of the process which can be denoted as

[TABLE]

where $n\in\left\{x\in\mathbb{N}\,:\;1\leq x<\log_{2}(T)-1\right\}$ therefore determines the number of consecutive observations considered for the average. If the process has constant mean $\mu$, this implies that $\bar{X}_{t}^{(n)}$ also has the same mean and, based on these averages and following [6], the maximum-overlapping AV (MOAV)

[TABLE]

where $m^{\star}=T-2n+1$ and whose corresponding estimator is given by

[TABLE]

where $\bar{x}_{t}^{(n)}$ denotes the sample equivalent of $\bar{X}_{t}^{(n)}$ based on a realization of the process $(X_{t})$. Another version of the AV is the non-overlapping AV (NOAV) whose estimator however is not statistically as efficient as that of the MOAV (see App. B for more details).

Keeping in mind the above definitions of the MOAV, [8] delivered a general theoretical form of this quantity when applied to weakly stationary processes which is given by

[TABLE]

Based on the above equation, the exact form of the AV for different stationary processes, such as the general class of AutoRegressive Moving Average (ARMA) models, can be derived. Moreover, [8] provided the theoretical AV for non-stationary processes such as the random walk and ARFIMA models for which the AV, as mentioned earlier, represents a better measure of uncertainty compared to other methods.

Using the known theoretical forms of the AV, it is therefore possible to detect and distinguish different processes based on the pattern of their AV. Due to this, this quantity (or similar quantities such as the Haar wavelet variance) can be used as a mean to estimate different kind of processes [see for example 4]. However, there are many commonly encountered processes whose AV is not known and it is unclear to what point these can actually be distinguished from stationary processes. The next section delivers a more general form of the AV which includes these processes and studies if this quantity can actually be helpful in detecting them.

III Allan Variance for Constant-Mean Non-Stationary Processes

As underlined in the previous sections, the AV is particularly useful for measuring uncertainty in non-stationary processes, especially when these have infinite variance. Nevertheless, there are other forms of non-stationarity for which the properties of the AV are unknown and these consist in those processes $(X_{t})$ with a constant mean $\mu$ (independent of time) but a non-stationary covariance structure. This implies that the covariance function between observations at distance $h$ is also a function of time $t$ and can therefore be denoted as $\gamma(h,t)$. This type of process is very common in different areas going from engineering [see 2] to economics [see 3].

To study the theoretical form of the AV for this class of processes, let us first define $\mathbf{X}_{t}^{(n)}$ as being the following vector of $n$ consecutive observations starting at $t-n+1$

[TABLE]

which contains the observations used to build the average in (1). Using the above vector, for $t=n,...,T$, we can then define the matrix

[TABLE]

and, for $t=2n,...,T$, we define the matrix

[TABLE]

These matrices represent the covariance matrices of the observations contained in each consecutive average. Indeed, $\bm{\Sigma}_{t}^{(n)}$ represents the covariance matrix of the observations within the average $\bar{X}_{t}^{(n)}$ which is used in definitions (D-1) and (2) while $\bm{\Gamma}_{t}^{(n)}$ represents the cross-covariance between these two sets of observations. A visual representation of these quantities is given in App. A.

In this section we will only consider the non-stationary MOAV while the form of the non-stationary NOAV is given in App. B. Based on the above matrices, we can also define different quantities according to the matrix of reference and the lags between observations. More specifically, let us first consider the case in which we are interested in lags $h$ such that $0\leq h<n$. Because of the overlapping nature of the AV, the observations at these lags can belong to the sets of observations within both the matrix $\bm{\Sigma}_{t}^{(n)}$ and the matrix $\bm{\Gamma}_{t}^{(n)}$ and, for these sets of observations within the matrix $\bm{\Sigma}_{t}^{(n)}$, we can define the following quantity

[TABLE]

If, however, the observations at the considered lags are among the set of observations only within the matrix $\bm{\Gamma}_{t}^{(n)}$, we define the quantity below

[TABLE]

Finally, when considering lags $h$ such that $n\leq h\leq 2n-1$, the set of observations at these lags can only be considered within the matrix $\bm{\Gamma}_{t}^{(n)}$ and, for this final case, we define the quantity

[TABLE]

The above definitions can be seen as generalized definitions of the autocovariance which consists in the average autocovariance for a given lag $h$. Because of this, it must be underlined that these definitions are not at all equivalent to the covariance function $\gamma(h,t)$ but correspond to an average of this function over all times $t$. Having specified this, we can now provide the following lemma.

Lemma 1:

The non-stationary MOAV is given by

[TABLE]

The proof of this lemma is given in App. C. Considering this expression, an aspect that must be underlined is that the definitions of the functions $\widetilde{\gamma}(h)$ and $\widetilde{\gamma}^{\ast}(h)$ given earlier simplify to the autocovariance function $\gamma(h)$ when dealing with a weakly stationary process. In the latter case, the form of the non-stationary AV consequently reduces to the expression in (II) for which a more detailed discussion is given in App. LABEL:app.statequiv. As a final note to this result it should also be highlighted that, in some of the considered non-stationary cases, the estimators of the MOAV defined in (3) and of the NOAV (see App. B) do not necessarily have the same expectation.

Having underlined these points and with the general definition of the AV given in Lemma 1, we can now investigate its properties when assuming the process of interest is within the class of processes treated in this paper. The next sections report some simulation studies regarding some of these processes in attempt to understand also whether the AV is a useful quantity to detect them and distinguish them from weakly stationary processes. In all cases, the simulated process is of length $T=1,000$ (except for the bias-instability process where $T=2,500$) and is simulated 50 times. The estimated AV is represented in plots along with the theoretical stationary and non-stationary forms of the AV in order to understand its behaviour under these different assumptions.

III-1 Non-Stationary White Noise

The first process we study is the non-stationary white noise by which we intend, without loss of generality, all those zero-mean processes whose variance changes with time. The evolution of the variance in time can either be completely random or can follow a specific parametric model such as, for example, a GARCH process. The goal of studying the AV for these types of processes is to understand whether it is able to detect such a structure in a time series and if it can be distinguished from a stationary white noise process. For this purpose, the true non-stationary process considered in the simulation study is generated from the following model

[TABLE]

where $\sigma_{t}^{2}=t$, with $t=1,\ldots,T$. The theoretical stationary form is based on the average of the variances used to simulate the processes (i.e. $\bar{\sigma}^{2}=\nicefrac{{(1+T)}}{{2}}$ in this example). Fig. 1 represents the estimated AVs along with the theoretical forms (stationary and non-stationary).

In this case, it can be seen how both theoretical forms correspond and the estimated AVs closely follow these quantities. This example confirms that the AV is therefore unable to distinguish between a stationary white noise process and a white noise process whose second-order behaviour is non-stationary.

III-2 Bias-Instability

The bias-instability process is a commonly known process in the engineering domain, specifically for inertial sensor calibration for navigation. The characteristic of this process is that it consists in different concatenated sequences (blocks) where, within each block, the realization of a random variable is repeated (i.e. constant). More formally, let $b_{i},\,\,i=1,\ldots,B$, represent the set of time indices belong to the $i^{th}$ block within the time series, and let $C_{i}\overset{iid}{\sim}\mathcal{N}(0,\sigma^{2})$. We can then define this process as

[TABLE]

One realization of the bias-instability process is illustrated in the top panel Fig. 2 where the length of block $b_{i}$ is 10 for all $i=1,\ldots,B$, and $B=250$.

Since the theoretical form of the AV for this process is not known exactly, it is often approximated by the AV of a First-Order AutoRegressive (AR1) process. Although this approximation can be useful, it is nevertheless still an approximation and, using the form given in Lemma 1, we can now obtain a theoretical form for the AV of this process which is represented in the bottom panel of Fig. 2. Indeed the latter plot shows that the estimated AVs closely follow the theoretical non-stationary form given earlier. The red line represents the AV of a stationary AR1 process which is supposed to approximate the true AV of bias-instability. The latter is the result of the averaging of the theoretical AV for a stationary AR1 process estimated via maximum-likelihood on each of the simulated processes. It is clear how, although close over some scales, this approximation is not good enough when considering the logarithmic representation of the AV. Therefore, knowing the exact form of the AV for this process would allow to better interpret the signals characterised by bias-instability.

III-3 Block-Structure Autoregressive Processes

As a final example we consider a block-structure AR1 process. Similarly to the bias-instability process, within this paper, we define a block-structure process as a process whose parameters are fixed but is made by concatenated time periods (blocks) where observations within each block are generated independently from those in the other blocks. An example is given by the settings of longitudinal studies in which each subject can be measured over time and, although the subjects are independent from each other, these measurements can be explained by an autocorrelated process within each subject. To define this process formally, let $X_{t}^{(i)}\sim F_{\bm{\theta}}$ denote the following AR1 process

[TABLE]

with parameter vector $\bm{\theta}=[\phi\,\,\,\,\sigma^{2}]^{T}$ where $\phi\in(-1,1)$ and $\epsilon_{t}\overset{iid}{\sim}\mathcal{N}(0,\sigma^{2})$. If again we let $b_{i}$ denote the $i^{th}$ block, then the block-structure AR1 process can be defined as

[TABLE]

where $X_{t}^{(i)}$ is independent of $X_{t}^{(j)}$, $\forall\,i\neq j$. By defining $\phi=0.9$ and $\sigma^{2}=1$ for the simulation study, the top panel of Fig. 3 shows a realization of this process while the bottom panel of Fig. 3 illustrates the results of the simulations for this particular process.

As for bias-instability, it can be observed how the stationary form of the AV (that does not consider the block structure) is not close to the estimated AVs while the non-stationary form provided in this paper adequately represents this process and can therefore allow to distinguish between a stationary autoregressive process and a block-structure one.

IV Conclusions

Within this paper we wanted to underline an issue concerning the AV which had not yet been studied. Indeed, the behaviour of the AV in commonly occurring settings where the covariance structure of the processes is non-stationary was unknown and, in many cases, was either ignored or dealt with through approximations. The consequence of the latter approaches would probably consist in erroneous interpretations and conclusions drawn from an AV analysis. For this reason, this paper studied the form of the AV for this class of processes thereby generalizing its form also for weakly stationary processes. Based on this, several examples were provided in which the properties of the AV were studied, highlighting its ability to detect these processes and to eventually distinguish them from stationary ones, making researchers and practitioners more aware of issues related to the interpretation and use of this quantity in more general and common settings.

Appendix A Graphical illustration of MOAV

To graphically illustrate the quantities defined in Section III, Fig. 4 represents the true covariance matrix for a given process and highlights how the AV is related to this matrix by overlapping square matrices along the diagonal, each of which is composed of the quantities defined in Eq. (5) and Eq. (6).

Appendix B Theoretical form of the NOAV for Non-Stationary Processes

The non-overlapping AV (NOAV) is defined as:

[TABLE]

where $m=\lfloor\frac{T}{2n}\rfloor$. The corresponding estimator for this quantity is given by

[TABLE]

This estimator is less efficient than the MOAV, mainly because it is based on fewer averages and therefore on a smaller sample size. To define the theoretical form of the NOAV for the non-stationary processes of interest, we first define the vector $\mathbf{X}_{j}^{(n)}$ of $n$ consecutive observations starting at $(j-1)n+1$, i.e.

[TABLE]

Using the above, for $k=1,...,m$, we define the matrices $\bm{\Sigma}_{2k}^{(n)}$, $\bm{\Sigma}_{2k-1}^{(n)}$ and $\bm{\Gamma}_{k}^{(n)}$ as follows:

[TABLE]

As in Appendix A, the above matrices are graphically represented in Fig. 5 where, as opposed to the MOAV, these matrices do not overlap along the diagonal of the covariance matrix of the process.

We then let $\bar{\sigma}^{(n)}_{2k}$, $\bar{\sigma}^{(n)}_{2k-1}$ and $\bar{\gamma}^{(n)}_{k}$ denote the averages of the matrices $\bm{\Sigma}_{2k}^{(n)}$, $\bm{\Sigma}_{2k-1}^{(n)}$ and $\bm{\Gamma}_{k}^{(n)}$, respectively, i.e.

[TABLE]

We further define $\bar{\sigma}^{(n)}$ and $\bar{\gamma}^{(n)}$ as follows:

[TABLE]

Based on the earlier defined matrices, as for the MOAV, we can also define different quantities according to the matrix of reference and the lags between observations. More specifically, let us first consider the case in which we are interested in lags $h$ such that $0\leq h<n$. The observations at these lags can belong to the sets of observations within the matrix $\bm{\Sigma}_{2k}^{(n)}$, $\bm{\Sigma}_{2k-1}^{(n)}$ and $\bm{\Gamma}_{k}^{(n)}$ and, for these sets of observations within matrices $\bm{\Sigma}_{2k}^{(n)}$ and $\bm{\Sigma}_{2k-1}^{(n)}$, we can define the following quantity

[TABLE]

If, however, the observations at the considered lags are among the set of observations only within the matrix $\bm{\Gamma}_{k}^{(n)}$, we define the quantity below

[TABLE]

Finally, when considering lags $h$ such that $n\leq h\leq 2n-1$, the set of observations at these lags can only be considered within the matrix $\bm{\Gamma}_{t}^{(n)}$ and, for this final case, we define the quantity

[TABLE]

As for the MOAV case, the above definitions can be seen as generalized definitions of the autocovariance which consists in the average autocovariance for a given lag $h$. Using the above notations and definitions, we can provide the following result.

Lemma 2:

We define the Non-stationary NOAV as

[TABLE]

Proof:

The proof of Lemma 2 is direct from the above definitions. Indeed, we have

[TABLE]

Then, using Eq. (D-1), we obtain

[TABLE]

which concludes the proof. ∎

Appendix C

Proof:

In order to prove Lemma 1 let $\bar{\sigma}^{(n)}_{t}$ and $\bar{\gamma}^{(n)}_{t}$ denote the averages of the matrices $\bm{\Sigma}_{t}^{(n)}$ and $\bm{\Gamma}_{t}^{(n)}$, respectively, i.e.

[TABLE]

Further, we define $\bar{\sigma}^{(n)}$ and $\bar{\gamma}^{(n)}$ as follows:

[TABLE]

Based on these definitions we have

[TABLE]

Then, using Eq. (2), we obtain

[TABLE]

which concludes the proof. ∎