The Age of Incorrect Information: A New Performance Metric for Status Updates

TL;DR

This paper introduces the Age of Incorrect Information (AoII), a new metric for status updates that emphasizes informative, correct, and fresh updates, and develops optimal policies for minimizing it under power constraints.

Contribution

The paper proposes AoII as a novel metric for status updates, formulates its optimization as a CMDP, and provides a low-complexity algorithm for optimal transmission policies under power constraints.

Findings

Always update policy minimizes average AoII, age, and error.

Optimal policies under power constraints are mixtures of Lagrange policies.

Simulation shows AoII outperforms AoI in maintaining informative updates.

Abstract

In this paper, we introduce a new performance metric in the framework of status updates that we will refer to as the Age of Incorrect Information (AoII). This new metric deals with the shortcomings of both the Age of Information (AoI) and the conventional error penalty functions as it neatly extends the notion of fresh updates to that of fresh "informative" updates. The word informative in this context refers to updates that bring new and correct information to the monitor side. After properly motivating the new metric, and with the aim of minimizing its average, we formulate a Markov Decision Process (MDP) in a transmitter-receiver pair scenario where packets are sent over an unreliable channel. We show that a simple "always update" policy minimizes the aforementioned average penalty along with the average age and prediction error. We then tackle the general, and more realistic case, where the transmitter cannot surpass a specific power budget. The problem is formulated as a Constrained Markov Decision Process (CMDP) for which we provide a Lagrangian approach to solve. After characterizing the optimal transmission policy of the Lagrangian problem, we provide a rigorous mathematical proof to showcase that a mixture of two Lagrange policies is optimal for the CMDP in question. Equipped with this, we provide a low complexity algorithm that finds the AoII-optimal operating point of the system in the constrained scenario. Lastly, simulation results are laid out to showcase the performance of the proposed policy and highlight the differences with the AoI framework.

Equations

$${\Delta }_{\_}\text{age}(t)=t-U(t),$$

$${\Delta }_{\_}\text{err}(t)=\text{mathbbm}1\{\hat{X}(t)\ne X(t)\},$$

$${\Delta }_{\_}\text{AoII}(t)=f(t)\times g(X(t),\hat{X}(t)),$$

$$g_{\_}\text{ind}(X(t),\hat{X}(t))=\text{mathbbm}{1}_{\_}\{X_{\_}t\ne {\hat{X}}_{\_}t\}.$$

$$g_{\_}\text{sq}(X(t),\hat{X}(t))=(X(t)-\hat{X}(t){)}^2.$$

$$g_{\_}\text{threshold}(X(t),\hat{X}(t))=\text{mathbbm}{1}_{\_}|X(t)-\hat{X}(t)|\ge c,$$

$$f_{\_}\text{linear}(t)=t-V(t).$$

$$f_{\_}\text{exponential}(t)=\exp (a(t-V(t))),$$

$$f_{\_}\text{threshold}(t)=\text{mathbbm}{1}_{\_}\{t-V(t)\ge d\},$$

$${\Delta }_{\_}prop(t)=f(t)\times g(X(t),\hat{X}(t))=(t-V(t))\text{mathbbm}1\{\hat{X}(t)\ne X(t)\}.$$

$$p_{\_}R+(N-1)p_{\_}t=1.$$

$$S(t)=(t-V(t))\text{mathbbm}1\{\hat{X}(t)\ne X(t)\},$$

$$\underset{\varphi \in \Phi }{\mathrm{minimize}}$$

$$\theta+V(S)=\min_{\_}{\psi\in\{0,1\}}\big{\{}S+\sum_{\_}{S^{\prime}\in\mathbb{N}}\Pr(S\rightarrow S^{\prime}|\psi)V(S^{\prime})\big{\}}\quad\forall S\in\mathbb{N}$$

$${\overline{C}}_{\_}AU=(N-1)p_{\_}t\frac{\frac{1}{(1-a{)}^2}}{1+\frac{(N-1)p_{\_}t}{1-a}}.$$

$${\overline{C}}_{\_}NU=\frac{(N-1)p_{\_}t}{(1-b{)}^2+(1-b)(N-1)p_{\_}t}.$$

$$\lim_{\_}{T\to+\infty}\text{sup}\>\frac{1}{T}\mathbb{E}^{\phi}\Big{(}\sum_{\_}{t=0}^{T-1}\delta\psi^{\phi}(t)\Big{)}\leq\delta_{\_}{\text{budget}},$$

$$\underset{\varphi \in \Phi }{\mathrm{minimize}}$$

$$\displaystyle\lim_{\_}{T\to+\infty}\text{sup}\>\frac{1}{T}\mathbb{E}^{\phi}\Big{(}\sum_{\_}{t=0}^{T-1}\psi^{\phi}(t)\Big{)}\leq\alpha.$$

$$f(\lambda,\phi)=\lim_{\_}{T\to+\infty}\text{sup}\>\frac{1}{T}\mathbb{E}^{\phi}\Big{(}\sum_{\_}{t=0}^{T-1}S^{\phi}(t)+\lambda\psi^{\phi}(t)|S(0)\Big{)}-\lambda\alpha.$$

$$\underset{\lambda \in {\mathbb{R}}^{+}}{\max }\underset{\varphi \in \Phi }{\min }f(\lambda ,\varphi ).$$

$$g(\lambda )=\underset{\varphi \in \Phi }{\min }f(\lambda ,\varphi ).$$

$$C(S(t),\psi (t))=S(t)+\lambda \psi (t).$$

$$\theta_{\_}1+V_{\_}1(S)=\min_{\_}{\psi\in\{0,1\}}\big{\{}S+\lambda\psi+\sum_{\_}{S^{\prime}\in\mathbb{N}}\Pr(S\rightarrow S^{\prime}|\psi)V_{\_}1(S^{\prime})\big{\}},$$

$$\underset{n\in \mathbb{N}}{\mathrm{minimize}}$$

$$\overline{C}(0,\lambda )=(N-1)p_{\_}t\frac{\frac{1}{(1-a{)}^2}}{1+\frac{(N-1)p_{\_}t}{1-a}}+\lambda (1-\alpha ),$$

$${\pi }_{\_}0(n)=\frac{1}{1+\frac{(N-1)p_{\_}t(1-b^n)}{1-b}+\frac{(N-1)p_{\_}ta{b}^{n-1}}{1-a}},$$

$${\pi }_{\_}k(n)=(N-1)p_{\_}t{b}^{k-1}{\pi }_{\_}01\le k\le n,$$

$${\pi }_{\_}k(n)=(N-1)p_{\_}t{b}^{n-1}{a}^{k-n}{\pi }_{\_}0k\ge n+1,$$

$$\overline{C}(n)=(N-1)p_{\_}t\frac{\frac{1+b^n(nb-n-1)}{(1-b{)}^2}+\frac{b^{n-1}a(n+\frac{1}{1-a})}{1-a}}{1+\frac{(N-1)p_{\_}t(1-b^n)}{1-b}+\frac{(N-1)p_{\_}ta{b}^{n-1}}{1-a}},$$

$${\overline{C}}_{\_}1(n,\lambda )=\lambda \frac{(N-1)p_{\_}t{b}^{n-1}}{(1-a)(1+\frac{(N-1)p_{\_}t(1-b^n)}{1-b}+\frac{(N-1)p_{\_}ta{b}^{n-1}}{1-a})}-\lambda \alpha .$$

Full text

I Introduction

With the proliferation of cheap sensors and devices, monitoring has become the new standard of technology applications. In these applications, a monitor is interested in having accurate information about a remote process (e.g., a car’s position and velocity [1], the humidity of a room[2], etc.). To achieve this goal, the transmitter side of the link sends time-stamped status updates over the network to maximize/minimize a specific performance metric. To address the shortcomings of the throughput and delay metrics in these types of scenarios, the Age of Information (AoI) has been introduced to capture the notion of information freshness. To that extent, the AoI quantifies the information time lag at the monitor. The motivation for such a framework is that having fresh knowledge about the process of interest should result in a better real-time estimation of the process. This prompted a surge of papers on the subject to explore the potentials of the metric mentioned above. Consequently, the AoI is now widely regarded as a fundamental performance measure in communication systems.

Since its establishment in [3], the efforts of researchers in the AoI area were divided on a wide variety of real-life scenarios that arise in communication networks. For example, the AoI was heavily studied in the framework of energy harvesting sources in [4]. Optimizing the average age in the case where the transmitter generates packets at will was considered in [5] where, interestingly, it was shown that a zero-wait policy is far from being optimal. The AoI metric has also been recently used as a performance metric in content caching [6]. Centralized scheduling with the goal of minimizing the average age has also captured a lot of research attention (e.g., [7, 8, 9, 10]). For instance, the optimization of the average AoI with hybrid ARQ under a resource constraint was studied in [11]. In another line of work, and as the AoI is of wide interest in sensors applications where devices are autonomous, distributed scheduling schemes were proposed in [12, 13, 14, 15]. For example, age-optimal back-off timers in CSMA environments were found in [15]. Since streams normally have different priority assignments, researchers have lately focused on studying the AoI in multi-class scenarios [16, 17, 18, 19, 20].

As seen above, most of the research in the AoI area has been heavily focused on calculating and optimizing the average AoI. However, as previously stated, the ultimate goal in the communication system in question is to have the best real-time remote estimation of the process of interest at the monitor side. This leads to the following important question: is the AoI really the perfect metric to be used to estimate in real-time a process remotely? There have been some recent efforts to try and answer this question. For example, it was shown in [21] that the optimal minimum Mean Squared Error (MSE) policy of a Wiener process over a channel with random delay is far from being age-optimal. This stems from the fact that the AoI, by definition, does not capture well the information content of the transmitted packets nor the current knowledge at the monitor. In fact, even when the monitor has perfect knowledge of the process in question, the AoI always increases with time and, therefore, an unnecessary penalty is paid. This basic observation showcases why the AoI may come short in this type of application. Similarly, the AoI was shown in [22] to be sub-optimal in minimizing the status error in remotely estimating Markovian sources. These observations prompted efforts to propose new performance metrics that deal with the shortcomings of the AoI. Among these efforts, a time-based metric dubbed as the Age of Synchronization (AoS) was introduced in the framework of content caching [23]. Specifically, the AoS is zero when the transmitter has no packets to send and it grows linearly with time when the transmitter side generates a new packet. Although the AoS includes the packets generation as a factor, it does not take into account the information structure of the source and the current estimate at the receiver, which limits its usage in remote estimation applications. In another work [24], the authors proposed different effective age metrics for which a lower effective age should undoubtedly lead to a lower prediction error. For example, the notion of Sampling Age was introduced and was defined as the age relative to an ideal sampling pattern $g(t)$ that minimizes the error. However, finding the optimal pattern $g(t)$ was deemed to be far from being trivial. As seen from the above efforts, the ultimate goal has been to propose new metrics or sampling/scheduling policies that minimize either the prediction error or the MSE. This raises a question of paramount importance: should the minimization of prediction error or mean squared error always be regarded as the definitive goal of the remote estimation scenario? To argue that this should not always be the case, we shed light on one of the shortcomings of these conventional error measures. The primary issue with these error functions is that they do not increasingly penalize the monitor for wrongfully estimating the process of interest. In other words, the same penalty is paid for being in an erroneous state no matter how long the monitor has been in it. To that extent, a monitor wrongfully thinking that a machine is at a normal temperature suffers from the same penalty no matter how long the machine has been overheating for. This clearly suggests that a more general framework should be introduced to deal with the shortcomings of these error measures.

In our paper, we pave the way for such a framework by introducing a new performance metric that deals with the above mentioned shortcomings of both the AoI and the error functions. To that end, we summarize in the following the key contributions of this paper:

• •

We first go into more depth on highlighting the shortcomings of the AoI and the error performance metrics in the case of remote process estimation. Aiming to deal with these shortcomings, we propose a new performance measure, which we will call the Age of Incorrect Information (AoII), that neatly extends the notion of fresh updates to that of fresh “informative” updates. The word informative refers to updates that bring new and correct information to the monitor side. This new measure also captures the deteriorating effect the wrong information can have with time on the system.

• •

Afterward, we focus on the case where a transmitter-receiver pair communicates over an unreliable channel. The transmitter sends status updates about an $N$ states Markovian information source with the goal of the receiver being to estimate it accurately. In this scenario, we aim to find the optimal transmission policy that minimizes the average proposed metric. By casting this problem into a Markov Decision Process (MDP), we show that in the case where no constraints on the power are imposed, an “always update” policy is able to minimize the average age, the prediction error, and the average AoII.

• •

Following that, we tackle the more realistic case where each transmission incurs a cost, and the transmitter has a power budget that cannot be surpassed. We cast our problem in this case into a Constrained Markov Decision Process (CMDP) that is known to be challenging to solve. To circumvent this difficulty, we provide a Lagrange approach that transforms the CMDP to an unconstrained MDP. The Lagrangian optimization problem is then thoroughly studied and structural results on its optimal policy are provided.

• •

Subsequently, we provide a rigorous mathematical proof to show that the optimal operating point of the CMDP is achieved by a mixture of two deterministic Lagrange policies. Similar results were established in the literature for the AoI optimization framework [11]. However, due to the inherent properties of the proposed AoII metric, the standard approach adopted in [11] cannot be followed. Specifically, the mathematical expressions involved in our case are not necessarily convex, which limits the applicability of the approach in [11]. Accordingly, we proceed in a different direction to establish the required results as will be seen in later sections of the paper. Armed with these results, we provide an algorithm that finds the AoII-optimal policy under the power constraint in logarithmic complexity.

• •

Lastly, we provide numerical implementations of our transmission policy that highlight its performance and showcase interesting insights on the differences between the AoI and the AoII frameworks.

The rest of the paper is organized as follows: Section II is dedicated to the motivation of the newly proposed framework. The system model, along with the dynamics of the proposed metric are presented in Section III. Section IV provides the MDP description of the problem along with its analysis in the unconstrained power scenario. In Section V, we thoroughly analyze the constrained scenario and propose an optimal approach to solve it. Numerical results that corroborate the theoretical findings are laid out in Section VI, while the paper is concluded in Section VII.

II Proposed metric

To put into perspective our line of work, we focus in this section on a particular scenario where a transmitter-receiver pair communicates. More specifically, the transmitter observes a process $X(t)$ and informs the receiver (monitor) about it by sending status updates over the network. Based on the last received update, the monitor constructs an estimate of the process, denoted by $\hat{X}(t)$. Time is considered to be discrete and normalized to the time slot duration. For simplicity, we suppose in this section that the process in question can only have two values $\{1,2\}$, as depicted in Fig 2. At each time slot, the probability of remaining in the same state is $p_{\_}R$ while the probability of transitioning to another state is $p_{\_}t$. The transmitter decides when to inform the monitor about the process $X(t)$ by adopting a transmission policy that aims to minimize the average of a particular penalty function.

First, let us consider the age penalty function to more closely examine its shortcomings. To that extent, we define the age as follows:

[TABLE]

where $U(t)$ is the time-stamp of the last successfully received packet by the monitor. Based on this definition, we can observe that the age captures the information time lag at the monitor, in an attempt to achieve timely updates. As seen in (1), the age always increases as time progresses regardless of the current information at the monitor, which makes it fall short in numerous applications. To see this, let us observe the trend of $\Delta_{\_}{\text{age}}(t)$ in the time interval $[0,t_{\_}1]$ of Fig. 1(a). In this interval, the monitor has perfect knowledge of the process of interest $X(t)$ and, therefore, any new update received in this interval will not change the information currently available at the monitor. Regardless of that, we can clearly see that the age penalty keeps growing with time, i.e., a penalty is being paid for not being updated on the information process although the monitor currently has perfect knowledge of the process in question. This above observation clearly put into perspective the shortcoming of the age penalty function and let us emphasize on the fact that any relevant metric for the remote estimation of a process has to capture more meaningfully its information content and the current knowledge at the receiver.

Another widely used penalty function is the error penalty:

[TABLE]

where $\mathbbm{1}$ is the indicator function. In fact, minimizing the average of the function in (2), is equivalent to the minimization of the prediction error $\Pr(\hat{X}(t)\neq X(t))$. The key shortcoming of this error penalty function is its failure to capture the following phenomena that arises in numerous applications: staying in an erroneous state should have an increasing penalty effect. In fact, the function in (2) treats all instances of error equally, no matter how long the time elapsed since their start is. In other words, the penalty of being in an erroneous state after $1$ time slot, or $100$ time slots is the same value of $1$. Because of this observation, we can see that the long-time average error penalty due to a burst error is the same as the one resulting from several isolated errors of the same duration. However, this is not always the case, and there exists a vast amount of applications where the penalty grows the longer the monitor has incorrect information. For example, let us suppose that $X(t)=1$ refers to the case where a machine is at a normal temperature at time $t$ while $X(t)=2$ is the case where the machine is overheating. This information has to be transferred to a monitor that can, consequently, react to the state of the machine. By considering the time interval $[t_{\_}1,t_{\_}2]$ of Fig. 1(b), we can see that no matter how long the duration of the interval $\Delta t=t_{\_}2-t_{\_}1$, the same penalty $\Delta_{\_}{err}(t)=1$ is kept. However, as it is well-known, the repercussions of keeping a machine overheated become more severe as time goes on. Therefore, this should be reflected in the adopted penalty function and should be considered as one of its key design features. It is worth mentioning that the list of such real life applications, where the level of dissatisfaction grows as time progresses, is vast. We report a few examples in the following:

• •

A real-time video stream in which packets are sent through a channel, and where losses can occur due to, for example, an inaccurate channel estimate. Similarly to the previous case, the adoption of the AoI as a performance metric will fall short since a penalty is constantly paid even if the current channel estimate is accurate. On another note, if any standard error penalty function is adopted, the effect of burst errors on the performance is not captured. However, it is well-known that in this application, a burst packet losses lead to more distortion of the video when compared to an equal number of isolated losses.

• •

An actuator that can tolerate inaccurate actions for a brief amount of time. However, when these actions are done for a long duration, substantial performance penalties are to be paid.

• •

The relay of fire outbreaks in environmental monitoring applications where any relay failure cause more severe repercussions the longer it lasts.

Motivated by all this, we aim to propose in our paper a new metric that elegantly combines the following two characteristics of the age and the error penalty functions:

The proposed metric captures the information content of the updates and the current knowledge of the monitor as done by the error penalty function in (2). 2. 2.

The proposed metric captures the increasing dissatisfaction with time that is offered by the age penalty.

Based on this, the general metric that we are about to introduce can be thought to capture the notion of fresh informative updates. The word informative in this context refers to updates that bring new information to the monitor side. In other words, when the monitor already has perfect knowledge about the process in question, we should not pay any penalty. However, as the state of the process change and the monitor becomes in an erroneous state, an update from the transmitter becomes informative. Because we need this update to arrive as fresh as possible, we let the penalty grows with time as long as we are in an erroneous state. To that extent, our proposed metric, which we will call the Age of Incorrect Information (AoII), can be written as follows:

[TABLE]

where $f(t)$ is an increasing time penalty function, paid for being unaware of the correct status of the process for a certain amount of time. On the other hand, $g(X(t),\hat{X}(t))$ is an information penalty function that reflects the difference between the current estimate at the monitor and the actual state of the process. There exists a wide variety of choices for $f$ and $g$ that we can pick from. We list below some of these examples, starting with $g$ and following it by $f$.

• •

The indicator error function:

[TABLE]

This information penalty function can be adopted when any mismatch between $X(t)$ and $\hat{X}(t)$ penalizes the system in the same fashion.

• •

The squared error function:

[TABLE]

Unlike $g_{\_}{\text{ind}}(X(t),\hat{X}(t))$, this information penalty function penalizes more the system the larger the difference between $X(t)$ and $\hat{X}(t)$ is.

• •

The threshold error function:

[TABLE]

where $c>0$ is a predefined threshold. This information penalty function can be used when the system can tolerate small mismatches between $X(t)$ and $\hat{X}(t)$. However, when the mismatch between the two is high, a penalty is paid.

Next, we provide examples of the time-dissatisfaction function $f$. To do so, we first define $V(t)$ as the last time instant where $g(X(t),\hat{X}(t))$ was equal to [math]. In other words, $V(t)$ is the last time instant where the monitor had zero information penalty, i.e., when the monitor had accurate information about the source. By leveraging this notion, we present the following examples of $f$.

• •

The linear time-dissatisfaction function:

[TABLE]

• •

The exponential time-dissatisfaction function:

[TABLE]

where $a>0$ is a positive constant. This time-dissatisfaction function can be used when the system is extremely vulnerable to wrong information and the need for fresh correct information grows quickly with time.

• •

The time-threshold dissatisfaction function:

[TABLE]

where $\mathbbm{1}_{\_}{\{.\}}$ is the indicator function, and $d>0$ is a fixed time threshold that should not be violated. This time-dissatisfaction function can be adopted when the system’s performance starts deteriorating due to wrong information beyond a certain time duration $d$.

For simplicity, we focus in the sequel on the case where $f(t)=f_{\_}{\text{linear}}$ and $g(X(t),\hat{X}(t))=g_{\_}{\text{ind}}$. Specifically, we have:

[TABLE]

A sketch of this function is given in Fig. 1(c) where we can see how the penalty increases as time progresses in the interval $[t_{\_}1,t_{\_}2]$ to reflect the increasing dissatisfaction of being in an erroneous state. This metric will be the basis of our analysis in the upcoming sections, where we aim to minimize its average in a general scenario of interest. With that in mind, we stress the fact that our proposed metric is far more general and is not limited to this choice of $f$ and $g$.

III System Overview

III-A System Model

We consider in our paper a transmitter-receiver pair where the transmitter sends status updates about the process of interest to the receiver side over an unreliable channel. Time is considered to be slotted and normalized to the slot duration (i.e., the slot duration is taken as $1$). The information process of interest is an $N$ states discrete Markov chain \big{(}X(t)\big{)}_{\_}{t\in\mathbb{N}} depicted in Fig. 3. To that extent, we define the probability of remaining at the same state in the next time echelon as $\Pr(X(t+1)=X(t))=p_{\_}R$. Similarly, the probability of transitioning to another state is defined as $\Pr(X(t+1)\neq X(t))=p_{\_}t$. Since the process in question can have one of $N$ different possible values, the following always holds:

[TABLE]

As for the unreliable channel model, we suppose that the channel realizations are independent and identically distributed (i.i.d.) over the time slots and follow a Bernoulli distribution. More precisely, the channel realization $h(t)$ is equal to $1$ if the packet is successfully decoded by the receiver side and is [math] otherwise. To that extent, we define the success probability as $\Pr(h(t)=1)=p_{\_}s$ and the failure probability as $\Pr(h(t)=0)=p_{\_}f=1-p_{\_}s$. We consider that when a packet is delivered to the receiver, the receiver sends an Acknowledgement (ACK) packet back to the transmitter. In the case of a failure of transmission, a negative-acknowledgment (NACK) is sent by the receiver. We suppose that the ACK/NACK packets are instantaneously delivered to the transmitter [5, 25]. This assumption is widely used in the literature since the ACK/NACK packets are small and, accordingly, their transmission times can be considered to be negligible. Using these ACK/NACK packets, the transmitter can have perfect knowledge of the information source estimate at the receiver at any time slot $t$.

The next aspect of our model that we tackle is the nature of packets in the system. To that extent, we consider that the transmitter can generate information updates any time at its own will. More specifically, when the transmitter decides to send an update at time $t$, it samples the process $X(t)$ and proceeds to the transmission stage. If the packet is not successfully delivered to the receiver, and if the transmitter desires a transmission retrial at time $t+1$, a new status update is generated by sampling $X(t+1)$ and the transmission stage begins again.

Lastly, and as previously explained in the preceding section, the transmitter’s ultimate objective is to adopt a transmission policy that minimizes the time average of a particular penalty function. In the sequel, we adopt the newly proposed metric reported in (10). To fully characterize it, we provide details on its dynamics in the next subsection.

III-B Penalty Function Dynamics

Let $S(t)$ be the penalty of the system mentioned above at time instant $t$. More specifically:

[TABLE]

where $V(t)$ is the last time instant where the monitor was in a correct state. In the sequel, we provide details concerning the dynamics of $S(t)$ in the aim of characterizing the values of $S(t+1)$. To do so, we first define $\psi(t)$ as the decision at time $t$ of the transmitter to either transmit (value $1$) or remain idle (value [math]). We distinguish in the following between two cases: $S(t)=0$ and $S(t)\neq 0$.

III-B1 $S(t)=0$

In this case, the monitor has perfect knowledge of the process of interest at time $t$. If the transmitter decides not to send a status update, then $S(t+1)$ will be equal to [math] if the process does not change value. This happens with a probability $p_{\_}R$. In the same fashion, $S(t+1)$ will be equal to $1$ if the process changes value, which happens with a probability $1-p_{\_}R=(N-1)p_{\_}t$. Let us now consider the case where the transmitter decides to send a status update at time $t$. Regardless of the channel realization, no new information will be conveyed to the monitor as $\hat{X}(t+1)$ will have the same value of $\hat{X}(t)$. Consequently, the previous analysis still holds for this case and $S(t+1)$ will be equal to [math] if the process does not change value and $1$ otherwise. We summarize what was stated in the following:

• •

\Pr\big{(}S(t+1)=0|S(t)=0,\psi(t)=0\big{)}=\Pr\big{(}S(t+1)=0|S(t)=0,\psi(t)=1\big{)}=p_{\_}R

• •

\Pr\big{(}S(t+1)=1|S(t)=0,\psi(t)=0\big{)}=\Pr\big{(}S(t+1)=1|S(t)=0,\psi(t)=1\big{)}=1-p_{\_}R=(N-1)p_{\_}t

III-B2 $S(t)\neq 0$

In this case, the monitor does not have correct knowledge of the process of interest (i.e., $\hat{X}(t)\neq X(t)$ ). If the transmitter decides to remain idle, then $S(t+1)$ will be equal to [math] if and only if the information process changes to the value that the monitor has from its last received update. More specifically, this is when $X(t+1)=X(U(t))$ with $U(t)$ being the time-stamp of the last successfully received packet by the monitor. This event occurs with a probability $p_{\_}t$. On the other hand, if the process keeps its same value, or transition to one of the remaining $N-2$ states, the penalty will grow by a step, i.e., $S(t+1)=S(t)+1$. Now, let us consider the case where the transmitter decides to send a packet. To that extent, we consider two cases:

• •

$h(t)=0$: In this case, the transmitted packet is not successfully decoded by the receiver. Therefore, no new knowledge is given to the monitor, i.e., $\hat{X}(t+1)=\hat{X}(t)$. To that extent, conditioned on $h(t)=0$, we can assert that $S(t+1)$ becomes zero if and only if the information process changes to the value that the monitor has from its last received update. As previously mentioned, this event occurs with a probability $p_{\_}t$. On the other hand, $S(t+1)$ will be equal $S(t)+1$ if the process keeps its same value or change to one of the other $N-2$ states, which happens with a probability $p_{\_}R+(N-2)p_{\_}t$.

• •

$h(t)=1$: In this case, the transmitted packet is successfully decoded by the receiver. Therefore, the estimate at the monitor $\hat{X}(t+1)$ is nothing but $X(t)$. To that extent, $S(t+1)$ will be equal to zero if the information process did not change during the transmission slot. This event happens with a probability $p_{\_}R$. On the other hand, if the process has changed during transmission to any of the remaining $N-1$ states, $S(t+1)$ will increase by $1$.

By taking into account the independence between the information process transitions and the channel realizations, we can summarize the transitions probabilities of $S(t)$ in the following:

• •

\Pr\big{(}S(t+1)=0|S(t)\neq 0,\psi(t)=0\big{)}=p_{\_}t

• •

\Pr\big{(}S(t+1)=S(t)+1|S(t)\neq 0,\psi(t)=0\big{)}=p_{\_}R+(N-2)p_{\_}t

• •

\Pr\big{(}S(t+1)=0|S(t)\neq 0,\psi(t)=1\big{)}=p_{\_}Rp_{\_}s+p_{\_}fp_{\_}t

• •

\Pr\big{(}S(t+1)=S(t)+1|S(t)\neq 0,\psi(t)=1\big{)}=p_{\_}Rp_{\_}f+(N-2)p_{\_}t+p_{\_}sp_{\_}t

IV Unconstrained Scenario

IV-A Problem Formulation

The objective of this paper is to find a transmission policy that minimizes the total average AoII of the network. A transmission policy $\phi$ is defined as a sequence of actions $\phi=(\psi^{\phi}(0),\psi^{\phi}(1),\ldots)$ where $\psi^{\phi}(t)=1$ if a transmission is initiated at time $t$. By letting $\Phi$ denote the set of all possible causal scheduling policies, our problem can be formulated as follows:

[TABLE]

IV-B MDP Characterization

Based on our model’s assumptions and the dynamics previously detailed in Section III-B, our problem in (13) can be cast into an infinite horizon average cost Markov decision process that is defined as follows:

• •

States: The state of the MDP at time $t$ is nothing but the penalty function $S(t)$. This penalty can have any value in $\mathbb{N}$. Therefore, the considered state space is countable and infinite.

• •

Actions: The action at time $t$, denoted by $\psi(t)$, indicates if a transmission is attempted (value $1$) or the transmitter remains idle (value [math]).

• •

Transitions probabilities: The transitions probabilities between the different states have been previously detailed in Section III-B.

• •

Cost: We let the instantaneous cost of the MDP, $C(S(t),\psi(t))$, to be simply the penalty function $S(t)$.

Finding the optimal solution of an infinite horizon average cost MDP is recognized to be challenging due to the curse of dimensionality. More precisely, it is well-known that the optimal policy $\phi^{*}$ of the problem mentioned above can be obtained by solving the following Bellman equation [26]:

[TABLE]

where $\Pr(S\rightarrow S^{\prime}|\psi)$ is the transition probability from state $S$ to $S^{\prime}$ given the action $\psi$, $\theta$ is the optimal value of (13) and $V(S)$ is the value function. Based on (14), one can see that the optimal policy $\phi^{*}$ depends on $V(.)$, for which there is no closed-form solution in general [26]. There exist various numerical algorithms in the literature that solve (14), such as the value iteration and the policy iteration algorithms. However, they suffer from being computationally demanding. To circumvent this complexity, we study in the next section the structural properties of the optimal transmission policy.

IV-C Structural Results

The first step in our structural analysis of the optimal policy consists of studying the particularity of the value function $V(.)$. To that extent, we provide the following lemma.

Lemma 1**.**

The value function $V(S)$ is increasing in $S$.

Proof:

The proof can be found in Appendix C of the supplementary material. ∎

The above lemma will be used in the following theorem to provide results on the optimal transmission policy.

Theorem 1**.**

The optimal transmission policy $\phi^{*}$ of our problem in (13) is:

• •

$p_{\_}t<p_{\_}R$: the transmitter should send updates at each time slot or when the receiver is in an erroneous state. In both cases, the optimal cost is:

[TABLE]

• •

$p_{\_}t\geq p_{\_}R$: it is optimal to never transmit any packet. In this case, the optimal cost is:

[TABLE]

with $a,b$ being two constants that are equal to $p_{\_}Rp_{\_}f+(N-2)p_{\_}t+p_{\_}sp_{\_}t$ and $p_{\_}R+(N-2)p_{\_}t$ respectively.

Proof:

The proof can be found in Appendix D of the supplementary material. ∎

The intuition behind the above results is that when $p_{\_}t<p_{\_}R$, a transmitted packet has a high chance of becoming erroneous by the time it is delivered to the receiver. Accordingly, in this case, the information source changes so fast to the point that transmitting packets will harm the performance of the system. As this case is not of practical interest, we focus in the rest of the paper on the scenario where $p_{\_}t<p_{\_}R$. Consequently, we have that in the case where no constraints on the power are imposed, the optimal minimum cost is achieved either by sending updates at every time slot or when the receiver is in an erroneous state.

Remark 1**.**

By adopting the same model as the one above, and by considering the AoI as the penalty function, we can verify by the same manners that the optimal transmission policy is to send updates at each time slot. As for the error penalty function, it can also be verified that sending updates at every time slot or when the receiver is in an erroneous case minimizes the prediction error. Consequently, and as the intuition suggests, an “always update” policy minimizes all the above $3$ penalties in the unconstrained power case. However, as will be shown in the sequel, this does not hold in the case of power-constrained scenarios.

V Power constrained scenario

V-A Problem Formulation

In realistic scenarios, a transmitter cannot send status updates at each time slot. In fact, each attempted transmission incurs a power cost $\delta$, and the transmitter has an average power budget $\delta_{\_}{\text{budget}}$ that cannot be surpassed. Consequently, the transmitter has to choose wisely when to transmit an update to the monitor as the following constraint has to be satisfied by any chosen transmission policy $\phi$:

[TABLE]

where the transmission policy $\phi$ is defined as a sequence of actions $\phi=(\psi^{\phi}(0),\psi^{\phi}(1),\ldots)$ such that $\psi^{\phi}(t)=1$ if a transmission is initiated at time $t$. Since \lim_{\_}{T\to+\infty}\text{sup}\>\frac{1}{T}\mathbb{E}^{\phi}\Big{(}\sum_{\_}{t=0}^{T-1}\psi^{\phi}(t)\Big{)}\leq 1, we define $\alpha=\frac{\delta_{\_}{\text{budget}}}{\delta}$ and we suppose that $\alpha\leq 1$ as the constraint becomes redundant otherwise. Putting it all together, our problem can be formulated as follows:

[TABLE]

To address the above problem, we proceed with a Lagrange approach that transforms our constrained minimization problem into an optimization of the Lagrangian function. More specifically, by letting $\lambda\in\mathbb{R}^{+}$ be the Lagrange multiplier, we define the Lagrangian function as follows:

[TABLE]

To that extent, the Lagrange approach can be summarized in the following problem:

[TABLE]

It is well-known that for any feasible scheduling policy $\phi$ satisfying the constraint in (18), the optimal value of the problem in (20) forms a lower bound to that of our original problem in (18). The difference between the two values is known as the duality gap, which is generally non-zero. Our goal is to show that our approach can achieve the optimal solution of the problem in (18). To that extent, we first study in the sequel the problem:

[TABLE]

V-B MDP Characterization

Similarly to the previous section, we cast the problem (21) into an MDP, which is the same as the one reported in the previous section except for the cost that is defined in this case as:

[TABLE]

Following the same line of work, we know that the optimal policy $\phi^{*}$ of the problem $\underset{\phi\in\Phi}{\text{min}}\>\>f(\lambda,\phi)$ can be obtained by solving the Bellman equation for all $S\in\mathbb{N}$:

[TABLE]

where $\Pr(S\rightarrow S^{\prime}|\psi)$ is the transition probability from state $S$ to $S^{\prime}$ given the action $\psi$, $\theta_{\_}1$ is the optimal value of the problem and $V_{\_}1(S)$ is the value function. As it was detailed in the previous section, solving the above equation directly is cumbersome in terms of complexity and hence, we provide structural properties of the optimal transmission policy in the next subsection.

V-C Structural Results

In the same spirit as the previous section, we start by investigating the particularity of the value function $V_{\_}1(.)$.

Lemma 2**.**

The value function $V_{\_}1(S)$ is increasing in $S$.

Proof:

The proof follows the same procedure of Lemma 1 and is therefore omitted for the sake of space. ∎

The above lemma will be used to show that the optimal policy of our problem is a threshold policy. Before providing the proof of our claim, we first lay out the following definition.

Definition 1**.**

An increasing threshold policy is a deterministic stationary policy in which the transmitter remains idle if the current state of the system $S$ is smaller than $n$ and attempts to transmit otherwise. In this case, the policy is fully characterized by the threshold $n\in\mathbb{N}$.

With the above definition being laid out, we present the following proposition.

Proposition 1**.**

The optimal policy $\phi^{*}$ of the problem in (21) is an increasing threshold policy.

Proof:

The proof can be found in Appendix E of the supplementary material. ∎

With the structure of the optimal policy of (21) being found, we tackle in more depth the average cost of our MDP when a threshold policy is adopted. To that extent, we recall that a threshold policy is fully characterized by its threshold value $n$. Accordingly, our problem in (21) can be reformulated as follows:

[TABLE]

where $\overline{C}(n,\lambda)$ is the infinite horizon average cost of the MDP when the threshold policy is adopted. To find the expression of $\overline{C}(n,\lambda)\>\>\forall n\in\mathbb{N}$, we first tackle the special case where the transmitter always send updates at each time slot (i.e., $n=0$). In this scenario, the portion of time where the transmitter is sending updates, which is defined as \lim_{\_}{T\to+\infty}\text{sup}\>\frac{1}{T}\mathbb{E}^{\phi}\Big{(}\sum_{\_}{t=0}^{T-1}\psi^{\phi}(t)\Big{)}, is equal to $1$. Moreover, by using Theorem 1, we end up with the following:

[TABLE]

with $a$ being equal to $p_{\_}Rp_{\_}f+(N-2)p_{\_}t+p_{\_}sp_{\_}t$. Next, we shift our attention to the case where $n\in\mathbb{N}^{*}$. To that extent, we note that for any threshold policy, the MDP can be modeled through a Discrete Time Markov Chain (DTMC) where:

• •

The states refer to the values of the penalty function $S(t)$.

• •

For any state $S(t)<n$, the transmitter is idle and therefore the dynamics of $S(t)$ coincide with those of $\psi(t)=0$ of Section III-B. On the other hand, for any state $S(t)\geq n$, the dynamics of $S(t)$ coincide with those of $\psi(t)=1$ of the same section.

Consequently, we focus in the sequel on this DTMC.

The next step towards finding the average cost $\overline{C}(n,\lambda)$ consists of calculating the stationary distribution of the DTMC. We, therefore, provide the following proposition.

Proposition 2**.**

For a fixed threshold $n\in\mathbb{N}^{*}$, the DTMC in question is irreducible and admits $\pi_{\_}k(n)\>\>\forall k\in\mathbb{N}$ as its stationary distribution where:

[TABLE]

[TABLE]

[TABLE]

with $a,b$ being two constants that are equal to $p_{\_}Rp_{\_}f+(N-2)p_{\_}t+p_{\_}sp_{\_}t$ and $p_{\_}R+(N-2)p_{\_}t$ respectively.

Proof:

The proof can be found in Appendix F of the supplementary material. ∎

By leveraging Proposition 2, we can proceed to find a closed form of the average cost of the threshold policy.

Theorem 2**.**

For a fixed threshold $n\in\mathbb{N}^{*}$, the average cost of the policy is $\overline{C}(n,\lambda)=\overline{C}(n)+\overline{C}_{\_}1(n,\lambda)$ where:

[TABLE]

[TABLE]

Proof:

The proof can be found in Appendix G of the supplementary material. ∎

As we now have the expression of the average cost $\overline{C}(n,\lambda)$, we turn our attention to studying its characteristics in order to prove the optimality of the Lagrange approach.

V-D Optimality of the Lagrange Approach

The optimality of the Lagrange approach in similar resource-constrained environments has been established in the literature for other cost functions (e.g., the AoI in [11]). However, contrary to [11], the standard approach to prove this optimality cannot be adopted in our case. This is mainly due to the complexity of the average cost function reported in Theorem 30. In particular, as seen in (29)-(30), $\overline{C}(n,\lambda)$ is not necessarily convex in $n$, which limits the applicability of the approach adopted in [11]. Accordingly, to demonstrate the optimality of the Lagrange approach in our case, we proceed in a different direction. Specifically, we investigate in more depth the behavior of the cost function and leverage these results to establish the AoII-optimal policy. To present our approach, we first let \big{(}A(n)\big{)}_{\_}{n\in\mathbb{N}} be the portion of time where the transmitter is attempting to send packets. To that extent, we have that \big{(}A(n)\big{)}_{\_}{n\in\mathbb{N}} is a decreasing positive sequence with $A(0)=1$ and \big{(}A(n)\big{)}_{\_}{n\in\mathbb{N}^{*}}=\sum\limits_{\_}{k=n}^{+\infty}\pi_{\_}k(n) which can be expressed as:

[TABLE]

To that end, we have $\overline{C}_{\_}1(n,\lambda)=\lambda A(n)-\lambda\alpha$. With this definition in mind, we summarize our approach in the following:

We prove that $\overline{C}(n)$, which is reported in (29), is increasing with $n$. 2. 2.

We define the set of intersection points

[TABLE] 3. 3.

We prove that $\lambda(n)$ is increasing with $n$. 4. 4.

We relate through graphical methods and several inductive lemmas the results on $\lambda(n)$ to the establishment of the AoII-optimal policy. 5. 5.

We propose a low complexity algorithm to find the AoII-optimal operating point of the system.

The details of the above steps will be laid out in the remainder of this section. To proceed in this direction, we also define $n(\lambda)$ as the optimum threshold that solves, for a fixed $\lambda$, the optimization problem in (24). With the definitions dealt with, and with our steps being clarified, we now proceed with the proof of optimality. To that extent, let us first note that the following always holds:

[TABLE]

where $\theta^{*}$ is the optimal value of our constrained problem in (18). Consequently, if we can find $\lambda_{\_}1$ such that $A(n(\lambda_{\_}1))=\alpha$, then $g(\lambda_{\_}1)=\underset{\lambda\in\mathbb{R}^{+}}{\text{max}}\>\>g(\lambda)=\theta^{*}$. In this case, we achieve the optimal operating point of (18) by simply adopting a threshold policy characterized by the threshold $n(\lambda_{\_}1)$. However, the issue arises when such a value of $\lambda_{\_}1$ does not exist since the set $\{n(\lambda):\>\lambda\in\mathbb{R}^{+}\}$ is discrete. To deal with this case, we aim to show that we can always find $(n_{\_}0,\lambda_{\_}{n_{\_}0})$ such that:

$\overline{C}(n_{\_}0,\lambda_{\_}{n_{\_}0})=\overline{C}(n_{\_}0+1,\lambda_{\_}{n_{\_}0})$ 2. 2.

$\begin{cases}A(n_{\_}0)\geq\alpha\\ A(n_{\_}0+1)<\alpha\end{cases}$ 3. 3.

$n(\lambda_{\_}{n_{\_}0})=n_{\_}0$

In this case, it is sufficient to take a mixture of two threshold policies $\phi_{\_}{n_{\_}0}$ and $\phi_{\_}{n_{\_}0+1}$ with a probability $\rho=\frac{\alpha-A(n_{\_}0+1)}{A(n_{\_}0)-A(n_{\_}0+1)}$ and $1-\rho=\frac{A(n_{\_}0)-\alpha}{A(n_{\_}0)-A(n_{\_}0+1)}$ respectively, to achieve the optimal objective value of the constrained problem in (18). We now proceed to show the existence and uniqueness of $(n_{\_}0,\lambda_{\_}{n_{\_}0})$.

Proposition 3**.**

The following always holds:

[TABLE]

Proof:

The proof can be found in Appendix A. ∎

As the above proposition holds for any $n$, let us focus on the value $n_{\_}0$ such that:

[TABLE]

In the next theorem, we show that this value $n_{\_}0$ verifies $n(\lambda_{\_}{n_{\_}0})=n_{\_}0$.

Theorem 3**.**

For the aforementioned $\lambda_{\_}{n_{\_}0}$, $n_{\_}0$ minimizes the average cost function $\overline{C}(n,\lambda_{\_}{n_{\_}0})$.

Proof:

The proof can be found in Appendix B. ∎

V-E Algorithm Implementation

Based on the previous section, we can assert that the optimal transmission policy consists of a mixture of two deterministic threshold policies $\phi_{\_}{n_{\_}0}$ and $\phi_{\_}{n_{\_}0+1}$ such that:

[TABLE]

As \big{(}A(n)\big{)}_{\_}{n\in\mathbb{N}} is a decreasing sequence in $n$, we can rewrite $n^{\prime}=n_{\_}0+1$ as follows:

[TABLE]

For any $0<\alpha\leq 1$, we can attest that there exists a finite $n^{\prime}$ that verifies the above condition. To find this value, we employ a two steps algorithm depicted in Algorithm $1$. The two steps are as follows:

• •

Exponential increase of the upperbound value $N_{\_}{UB}$ to ensure that $n^{\prime}$ is included in the interval of interest $[N_{\_}{LB},N_{\_}{UB}]$.

• •

A binary search in the interval mentioned above to find the value $n^{\prime}$.

The first part of the algorithm finishes in $N_{\_}1=\log_{\_}{2}(n^{\prime})$ iterations while the binary search part is known to have a worst-case complexity of $\log_{\_}2(N_{\_}{size})$ where $N_{\_}{size}$ is the size of the interval of interest. To that extent, we have that: $N_{\_}{size}=2^{N_{\_}1}-2^{N_{\_}1-1}$. Hence, the worst-case complexity of the second part is $\log_{\_}2(N_{\_}{size})=N_{\_}1-1$. We can, therefore, conclude that the complexity of the above algorithm is logarithmic in the value of $n^{\prime}$, which makes it appealing to be implemented in practice.

After the algorithm finishes and $n_{\_}0$ is found, it is sufficient to adopt a transmission policy where a packet is generated and transmitted when the penalty is equal to $n_{\_}0$ and $n_{\_}0+1$ with a probability $\rho=\frac{\alpha-A(n_{\_}0+1)}{A(n_{\_}0)-A(n_{\_}0+1)}$ and $1-\rho=\frac{A(n_{\_}0)-\alpha}{A(n_{\_}0)-A(n_{\_}0+1)}$ respectively to achieve the optimal objective value of the constrained problem in (18).

VI Numerical Results

In this section, we provide numerical results that highlight the effects of the information source dynamics on the performance of our proposed AoII-optimal policy. We also compare our framework to both the AoI and the error function minimization frameworks in order to shed light on important insights. Note that, although we focus on the Markovian information source depicted in Section III-A, the insights provided in this section intuitively hold for more general information source models.

VI-A Information Source Parameters

In the first scenario, we investigate in more depth the effect of the Markov chain’s dynamics on the performance of our proposed AoII-optimal policy.

VI-A1 Effect of $p_{\_}R$

In this scenario, we consider that the number of states is $N=8$, and we fix the parameter $\alpha$ to $0.1$. As for the channel parameter, we assume that the transmission success probability $p_{\_}s$ is equal to $0.8$. While making sure that $p_{\_}t<p_{\_}R$, we vary the probability of remaining in the same state $p_{\_}R$ and plot the average AoII of the optimal policy. As seen in Fig. 5, the average cost decreases as $p_{\_}R$ increases. The reason behind this is twofold:

When $p_{\_}R$ is high, the information source becomes more “predictable”. In other words, when a packet is transmitted, it is less likely for it to become obsolete due to a transition of the Markov chain during the transmission stage. 2. 2.

When $p_{\_}R$ is high, the AoII remains zero for a significant amount of time upon successful transmission. This allows us to make better use of the permitted power budget $\alpha$ as we will be able to transmit at a lower threshold value without exceeding the allowed power budget. This can be verified by looking at $n_{\_}0$ in function of $p_{\_}R$ in the following table:

[TABLE]

We can see from the above table that as $p_{\_}R$ increases, the value of $n_{\_}0$ decreases. In other words, our tolerance for the value of the AoII is reduced, and we can transmit at a much lower AoII value without violating the power constraint. This eventually leads to a reduction in the average AoII.

VI-A2 Effect of $N$

We consider the case where $p_{\_}R=0.5$, $\alpha=0.1$, and the probability of successful transmission is $p_{\_}s=0.8$. We vary $N$ and report the average AoII when the AoII-optimal policy is adopted in Fig. 6. As can be seen in the figure, the average AoII increases when the number of states $N$ grows. To explain this trend, we first recall that the transition probabilities at each state always verify the following equality:

[TABLE]

Accordingly, we can use (38) to conclude that $p_{\_}t=\frac{1-p_{\_}R}{N-1}$. Next, let us consider that the monitor has perfect knowledge of the information process at time $t$, denoted by $X(t)$. Then, let us suppose that the information source changes value at time $t+1$, which happens with a fixed probability $1-p_{\_}R$. With that in mind, we recall that the probability for the information source to go back to its old value $X(t)$ at time $t+2$ is $p_{\_}t$. As $p_{\_}t$ is a decreasing function in $N$, this means that the probability for the monitor to have correct knowledge of the information source at time $t+2$ without wasting resources for packet transmission, decreases with $N$. Accordingly, when $N$ grows, the average AoII will also increase.

VI-B Comparison with the AoI Framework

In the following, we provide a comparison between our optimal transmission policy and the optimal age policy of [11].

VI-B1 Comparison in Function of $\alpha$

We adopt in this case the same number of states $N=8$ and success probability $p_{\_}s=0.8$. We fix the probability of remaining in the same state $p_{\_}R$ to $0.5$. We vary the parameter $\alpha$ and plot the average AoII achieved by both policies. As seen in Fig. 7, the proposed policy always outperforms the age-optimal policy for all values of $\alpha$. The following two observations can also be drawn from the figure:

One can see that the two curves converge as $\alpha$ increases. This is in agreement with our theoretical results in the unconstrained case in Section IV. In fact, when the imposed power constraint becomes less restrictive, the transmitter will be sending more packets and we converge to the “always update” policy that minimizes both the AoII and the AoI. 2. 2.

Another interesting observation is that the gap between the two curves is small when $\alpha$ is small (e.g., the gap is equal to $1.1$ for $\alpha=0.02$). This is due to the number of packets sent by the transmitter becoming very small. Consequently, the average AoII will be mostly dictated by how the Markov chain evolves rather than the transmission policy adopted. Therefore, in this case, we converge to the “no updates” average cost previously reported in eq. (16).

By combining the above two observations, we can conclude that when the transmitter is heavily constrained by its power, or when it has unlimited power, age-optimal policies lead to virtually the same performance as the optimal AoII policy.

We also investigate the age performance of our proposed policy and compare it to the age-optimal policy. As seen in Fig. 8, the age-optimal policy outperforms our policy in terms of average age. However, the gap between the two curves vanishes for high $\alpha$ and that is for the same reason previously reported in the average AoII comparison between the two policies. On the other hand, as $\alpha$ decreases, the gap between the two curves increases, reaching $190$ for $\alpha=0.02$. The reason behind this is the fact that as $\alpha$ decreases, the allowed number of transmissions becomes extremely small. Therefore, the impact of the transmission decisions will become more significant on the performance. To that extent, since our policy is based on the information content of the packet rather than just the age at the monitor, our proposed penalty measure can sometimes be equal to [math] while the age is equal to $100$. The differences of spirit between the two transmission polices will lead to a significant difference in age performance when the available power budget is really small.

VI-B2 Comparison in Function of $p_{\_}R$

In this scenario, we compare the AoII-optimal policy and the AoI-optimal policy when $p_{\_}R$ is varied. We consider that $N=8$, $\alpha=0.1$, and the probability of successful transmission is $p_{\_}s=0.8$. While maintaining $p_{\_}t<p_{\_}R$, we vary $p_{\_}R$ and report the differences between the two policies in Fig. 9. As can be seen in the figure, the gap between the two curves increases as $p_{\_}R$ grows (from $0.7$ for $p_{\_}R=0.2$ to $2.2$ for $p_{\_}R=0.9$). To explain this, we recall that the AoI always increases regardless of the value of the information source. As $p_{\_}R$ increases, the information process $X(t)$ will have a higher probability of keeping the same value at the next time slot $t+1$. However, since the AoI is always increasing, the AoI-optimal policy will waste vital resources to update the monitor when it is not necessary to do so. As the AoI-optimal policy sends more obsolete packets when $p_{\_}R$ is high, this will to a non-negligible gap between the AoI-optimal and AoII-optimal policies as seen in the figure.

VI-C Comparison with the Error Framework

We present in the following a comparison between our policy and the error-based policy that follows the rules below:

• •

Send a packet solely when the monitor has a wrong estimate of the information source.

• •

Ensure that the constraint on the power consumption is verified. with equality.

We consider the case where $N=8$, $p_{\_}R=0.5$, and the probability of successful transmission is $p_{\_}s=0.8$. We report in the next table the AoII values of the two policies. As can be seen in Table II, our policy always outperforms the error based policy. We can also see that as $\alpha$ increases, the gap between the two shrinks since the transmitter will be sending more packets and we converge to the “always update” policy that minimizes both the AoII and the status error function.

[TABLE]

VII Conclusion and Future Work

In this paper, we have proposed a new performance metric that deals with the shortcomings of the conventional AoI and error penalty functions in the framework of status updates. Dubbed as the Age of Incorrect Information, this new metric extends the notion of fresh updates and adequately captures the information content that the updates bring to the monitor. We have studied the metric mentioned above in the case where a transmitter-receiver pair communicates over an unreliable channel. By leveraging MDP tools, the optimal policy’s structure was found for the cases where the transmitter is limited and non-limited by its power. A low complexity algorithm was then presented that finds the optimal operating point that minimizes the average AoII. Lastly, numerical results were laid out that highlight the effect of the information source’s dynamics on the AoII, along with a comparison between the AoI and AoII frameworks. The analysis in this paper can be used as a basis for the multi-user case, similar to the work done in [10]. Other future research directions include the extension to more general information source models, the investigation of broader choices of time-dissatisfaction and error functions, and the examination of continuous-time systems.

Appendix A Proof of Proposition 34

Before investigating the general scenario, we first note that the proposition is trivially true for $n=0$. In fact, we first note that $\overline{C}(n)$ is nothing but the average penalty of a threshold policy in the unconstrained MDP case reported in Section IV-B. As $\overline{C}(0)=\overline{C}(1)$ (we refer the readers to the results of Theorem 1), we can easily verify that we have $\overline{C}(0,0)=\overline{C}(1,0)$. To tackle the case where $n\in\mathbb{N}^{*}$, we provide a proof that revolves around a graphical illustration in Fig. 10 of $\overline{C}(n,\lambda)$ in function of $\lambda$. To proceed in that direction, we first study in the next lemma the variation of $\overline{C}(n)$ in function of $n,\forall n\in\mathbb{N}^{*}$.

Lemma 3**.**

The function $\overline{C}(n)$ is increasing with $n$.

Proof:

By considering the expression of $\overline{C}(n)$ previously reported in (29), we can observe that it is rather difficult to study its variations directly. To circumvent this difficulty, we recall that $\overline{C}(n)$ is nothing but the average penalty of a threshold policy in the unconstrained MDP case reported in Section IV-B. The dynamics of such a threshold policy is identical to the DTMC reported in Fig. 4. By observing the DTMC in question, we can see that the chain can only move backward due to a transition to state [math]. When the transmitter does not attempt to send a packet ($S<n$), the probability of transition to state [math] is $p_{\_}t$. However, when the transmitter sends packets ($S\geq n$), the probability of reducing the penalty to zero is $p_{\_}fp_{\_}t+p_{\_}Rp_{\_}s$. As $p_{\_}R>p_{\_}t$, we can conclude that $p_{\_}fp_{\_}t+p_{\_}Rp_{\_}s>p_{\_}t$. Consequently, a transmission of a packet will always increase the likelihood of transitions to the state [math]. Based on this, we can conclude that employing a higher threshold, which leads to a smaller number of transmissions, will undoubtedly increase the average penalty. ∎

By using the above results, and as $\overline{C}(n,0)=\overline{C}(n)$, we can conclude that the points on the $y-$axis in Fig. 10 move upwards as $n$ increases. Moreover, by using the expression of $\overline{C}(n,\lambda)$ in Theorem 30, we can deduce that the slope of $\overline{C}(n,\lambda)$ is nothing but $A(n)-\alpha$. Since $A(n)$ decreases when the threshold $n$ increases, we can assert that the slope of the curves $\overline{C}(n,\lambda)$ decreases with $n$. By combining the above two observations, we can see that for any fixed value of $n$, the two curves $\overline{C}(n,\lambda)$ and $\overline{C}(n+1,\lambda)$ intersect at a unique point $\lambda_{\_}0$.

Appendix B Proof of Theorem 3

To show that $n(\lambda_{\_}{n_{\_}0})=n_{\_}0$, it is sufficient to show that for any $n\neq n_{\_}0$, we have that $\overline{C}(n,\lambda_{\_}{n_{\_}0})\geq\overline{C}(n_{\_}0,\lambda_{\_}{n_{\_}0})$. To prove this, the first step of our analysis consists of studying the behavior of the intersection points $\lambda(n)$ as $n$ increases. More precisely, we consider the sequence \big{(}\lambda(n)\big{)}_{\_}{n\in\mathbb{N}} as the intersection point between $\overline{C}(n,\lambda)$ and $\overline{C}(n+1,\lambda)$. By using the definition in (34), we have that:

[TABLE]

To pursue our analysis, we provide key results on the behavior of the intersection points in the following proposition.

Proposition 4**.**

The sequence \big{(}\lambda(n)\big{)}_{\_}{n\in\mathbb{N}} is increasing with $n$.

Proof:

As a first step in the proof, we recall that due to the results of Lemma 3 and the decreasing nature of $A(n)$, we have that $\lambda(n)\geq 0\>\>\forall n\in\mathbb{N}$. As $\overline{C}(0)=\overline{C}(1)$, we can deduce that $\lambda(0)=0$ and therefore, we can restrict ourselves to study the increasing property of \big{(}\lambda(n)\big{)}_{\_}{n\in\mathbb{N}} solely for the case where $n\in\mathbb{N}^{*}$. To that extent, as seen in Theorem 30, the expression of the average cost is far from trivial. Consequently, to be able to study the variations of \big{(}\lambda(n)\big{)}_{\_}{n\in\mathbb{N}^{*}}, we first provide a lemma that will be useful to our analysis.

Lemma 4**.**

The series \big{(}\pi_{\_}0(n)\big{)}_{\_}{n\in\mathbb{N}^{*}} is decreasing with $n$.

Proof:

To prove this, let us consider the series $h(n)=\frac{1}{\pi_{\_}0(n+1)}-\frac{1}{\pi_{\_}0(n)}\>\>\forall n\in\mathbb{N}^{*}$. By replacing $\pi_{\_}0(n)$ and $\pi_{\_}0(n+1)$ by their respective values, we can show that:

[TABLE]

In other words, the series $h(n)\frac{(1-a)}{(N-1)p_{\_}t(b-a)}$ is a geometric series with a common ratio $b$. As $a<1$, we can conclude that the sign of $h(n)$ depends on the sign of $b-a$. To that extent, and by keeping in mind that $p_{\_}t<p_{\_}R$, we have that $b-a=p_{\_}R(1-p_{\_}f)-p_{\_}sp_{\_}t=p_{\_}s(p_{\_}R-p_{\_}t)>0$. Hence, we can conclude that $h(n)=\frac{\pi_{\_}0(n)-\pi_{\_}0(n+1)}{\pi_{\_}0(n)\pi_{\_}0(n+1)}\geq 0\>\>\forall n\in\mathbb{N}^{*}$. Baring in mind that $\pi_{\_}0(n)\geq 0\>\>\forall n\in\mathbb{N}^{*}$, we can assert that $\pi_{\_}0(n)\geq\pi_{\_}0(n+1)\>\>\forall n\in\mathbb{N}^{*}$ which concludes our proof. ∎

With the above lemma being laid out, we now find an explicit expression of the following difference: $\Delta\overline{C}=\overline{C}(n+1)-\overline{C}(n)$. As we have previously mentioned, the expression of the average cost is complicated, which makes treating the difference $\Delta\overline{C}$ a challenging task. To that extent, we provide in the following the $8$ terms that make up $\Delta\overline{C}$:

• •

z_{\_}1=\frac{(N-1)p_{\_}t\big{(}\pi_{\_}0(n+1)-\pi_{\_}0(n)\big{)}}{(1-b)^{2}}

• •

z_{\_}2=\frac{(N-1)p_{\_}tnb^{2}\big{(}b^{n}\pi_{\_}0(n+1)-b^{n-1}\pi_{\_}0(n)\big{)}}{(1-b)^{2}}

• •

z_{\_}3=\frac{(N-1)p_{\_}tnb\big{(}-b^{n}\pi_{\_}0(n+1)+b^{n-1}\pi_{\_}0(n)\big{)}}{(1-b)^{2}}

• •

$z_{\_}4=\frac{(N-1)p_{\_}tb^{n+1}(b-1)\pi_{\_}0(n+1)}{(1-b)^{2}}$

• •

z_{\_}5=\frac{(N-1)p_{\_}tan\big{(}b^{n}\pi_{\_}0(n+1)-b^{n-1}\pi_{\_}0(n)\big{)}}{1-a}

• •

$z_{\_}6=\frac{(N-1)p_{\_}tab^{n}\pi_{\_}0(n+1)}{(1-a)}$

• •

z_{\_}7=\frac{(N-1)p_{\_}t\big{(}-b^{n+1}\pi_{\_}0(n+1)+b^{n}\pi_{\_}0(n)\big{)}}{(1-b)^{2}}

• •

z_{\_}8=\frac{(N-1)p_{\_}ta\big{(}b^{n}\pi_{\_}0(n+1)-b^{n-1}\pi_{\_}0(n)\big{)}}{(1-a)^{2}}

Next, we divide each term by the expression $A(n)-A(n+1)$ previously reported in Section V-D. By replacing the terms with their values, and after algebraic manipulations, we can verify that the terms that constitute the expression of $\lambda(n)$ are:

• •

$g_{\_}1=\frac{z_{\_}1}{A(n)-A(n+1)}=\frac{(1-a)(-b(N-1)p_{\_}t+\frac{(N-1)p_{\_}ta(1-b)}{1-a})}{(1-b)^{3}(1+\frac{(N-1)p_{\_}t}{1-b})}$

• •

$g(n)=\frac{\sum\limits_{\_}{i=2}^{6}z_{\_}i}{A(n)-A(n+1)}=\frac{b-a}{1-b}(n-\frac{b}{\frac{\pi_{\_}0(n)}{\pi_{\_}0(n+1)}-b})$

• •

$g_{\_}7=\frac{z_{\_}7}{A(n)-A(n+1)}=\frac{b(1-a)}{(1-b)^{2}}$

• •

$g_{\_}8=\frac{z_{\_}8}{A(n)-A(n+1)}=\frac{-a}{1-a}$

We can see that $g_{\_}1$, $g_{\_}7$, and $g_{\_}8$ are only constant terms. On the other hand, the term $g(n)$ requires further investigation. To that extent, we provide the following lemma.

Lemma 5**.**

The series \big{(}g(n)\big{)}_{\_}{n\in\mathbb{N}^{*}} is increasing with $n$.

Proof:

First of all, let us define the ratio $r(n)$ as $\frac{\pi_{\_}0(n)}{\pi_{\_}0(n+1)}$. To study the variations of \big{(}g(n)\big{)}_{\_}{n\in\mathbb{N}^{*}}, we consider the difference $\Delta g(n)=g(n+1)-g(n)$. By using the expression of $g(n)$, we have that:

[TABLE]

As $r(n)\geq 1\geq b\>\>\forall n\in\mathbb{N}^{*}$ (we recall the results of Lemma 4), we can conclude that it is enough to study the sign of the numerator in (41). By replacing $r(n)$ with its expression, we can see that to prove $\Delta g(n)\geq 0$, it is sufficient to have:

[TABLE]

By replacing $\pi_{\_}0(n),\pi_{\_}0(n+1)$ and $\pi_{\_}0(n+2)$ with their expressions using (26), we can show that the LHS of (42) becomes $-(b-1)^{2}(1+\frac{(N-1)p_{\_}t}{1-b})$ which is always negative since $b\leq 1$. Therefore, we have that \big{(}g(n)\big{)}_{\_}{n\in\mathbb{N}^{*}} is an increasing sequence with $n$. ∎

From the above lemma, we can conclude that $\lambda(n)$ is the sum of two terms: a constant and an increasing function with $n$. Therefore, the sequence \big{(}\lambda(n)\big{)}_{\_}{n\in\mathbb{N}^{*}} is increasing with $n$ which concludes our proof. ∎

Our subsequent analysis will be divided into two sections where we study the thresholds $n$ that are larger than $n_{\_}0$ and prove that they lead to a cost $\overline{C}(n,\lambda_{\_}{n_{\_}0})$ that is higher than $\overline{C}(n_{\_}0,\lambda_{\_}{n_{\_}0})$. The case where $n<n_{\_}0$ is then tackled in the section after it.

B-1 $n>n_{\_}0$

To analyze this case, we first provide the following lemma.

Lemma 6**.**

$\forall k_{\_}2>k_{\_}1$, we consider two sequences \big{(}U_{\_}1(n)\big{)}_{\_}{n\in\mathbb{N}} and \big{(}U_{\_}2(n)\big{)}_{\_}{n\in\mathbb{N}} such that \big{(}U_{\_}2(n)\big{)}_{\_}{n\in\mathbb{N}^{*}} is an increasing sequence. If $\frac{U_{\_}1(n+1)-U_{\_}1(n)}{U_{\_}2(n+1)-U_{\_}2(n)}$ increases with $n$, then the following holds:

[TABLE]

Proof:

The proof is based on mathematical induction. More precisely, we know that the above lemma is true for $k_{\_}2=k_{\_}1+1$. We suppose that it is true for any $k_{\_}2>k_{\_}1+1$ and investigate the property for $k_{\_}2+1$. To that extent, we have that $\frac{U_{\_}1(k_{\_}2+1)-U_{\_}1(k_{\_}1)}{U_{\_}2(k_{\_}2+1)-U_{\_}2(k_{\_}1)}$ can be rewritten as:

[TABLE]

By multiplying the first and second term by $\frac{U_{\_}2(k_{\_}2+1)-U_{\_}2(k_{\_}2)}{U_{\_}2(k_{\_}2+1)-U_{\_}2(k_{\_}2)}$ and $\frac{U_{\_}2(k_{\_}2)-U_{\_}2(k_{\_}1)}{U_{\_}2(k_{\_}2)-U_{\_}2(k_{\_}1)}$ respectively, and by taking into account the increasing property of the ratio $\frac{U_{\_}1(n+1)-U_{\_}1(n)}{U_{\_}2(n+1)-U_{\_}2(n)}$ along with the induction assumption, the results can be found to be true for $k_{\_}2+1$ which concludes our proof. ∎

We can apply the above lemma by taking $U_{\_}1(n)=\overline{C}(n)$, $U_{\_}2(n)=-A(n)$ and noting the results of Proposition 4 on \big{(}\lambda(n)\big{)}_{\_}{n\in\mathbb{N}}. Consequently, Lemma 43 tell us that the intersection between $\overline{C}(n,\lambda)$ and $\overline{C}(n_{\_}0,\lambda)$ for any $n>n_{\_}0+1$ occur after $\lambda_{\_}{n_{\_}0}$. By observing Fig. 11, we can see that this leads to $\overline{C}(n,\lambda_{\_}{n_{\_}0})$ being larger than $\overline{C}(n_{\_}0,\lambda_{\_}{n_{\_}0})$ due to the properties of the curve $\overline{C}(n,\lambda)$ previously reported in Lemma 3.

B-2 $n<n_{\_}0$

Similarly to the previous subsection, we provide two vital lemmas to our analysis.

Lemma 7**.**

$\forall n\geq 1$, if the conditions of Lemma 43 are satisfied, we have that:

[TABLE]

[TABLE]

Proof:

We first start by rewriting $\frac{U_{\_}1(n+1)-U_{\_}1(n-1)}{U_{\_}2(n+1)-U_{\_}2(n-1)}$ as $\frac{U_{\_}1(n+1)-U_{\_}1(n)}{U_{\_}2(n+1)-U_{\_}2(n-1)}+\frac{U_{\_}1(n)-U_{\_}1(n-1)}{U_{\_}2(n+1)-U_{\_}2(n-1)}$. Afterward, the proof is based on multiplying the above expression by $\frac{U_{\_}2(n+1)-U_{\_}2(n)}{U_{\_}2(n+1)-U_{\_}2(n)}$ and $\frac{U_{\_}2(n)-U_{\_}2(n-1)}{U_{\_}2(n)-U_{\_}2(n-1)}$ and using the conditions of the lemma to prove the LHS and RHS inequalities, respectively. The details are omitted for the sake of space. ∎

Lemma 8**.**

$\forall n\leq n_{\_}0-1$, we always have that:

[TABLE]

Proof:

The proof is based on a mathematical backward induction. As a first step, we tackle the case for $n=n_{\_}0-1$. As $\lambda(n)$ is increasing with $n$, we have that $\frac{\overline{C}(n_{\_}0)-\overline{C}(n_{\_}0-1)}{A(n_{\_}0-1)-A(n_{\_}0)}\geq\frac{\overline{C}(n_{\_}0-1)-\overline{C}(n_{\_}0-2)}{A(n_{\_}0-2)-A(n_{\_}0-1)}$. By applying Lemma 46 for $n=n_{\_}0-1$, we can conclude that the above property is true for $n=n_{\_}0-1$. We now suppose that this property holds for any $n<n_{\_}0-1$ and aim to prove it to be true for $n-1$. By using our supposition, along with the increasing property of $\lambda(n)$ and the results of Lemma 46, the property can be verified to be true for $n-1$ which concludes our proof. ∎

Equipped with the above two lemmas, we will be able to show that for any $n<n_{\_}0$, we have that $\overline{C}(n_{\_}0,\lambda_{\_}{n_{\_}0})\leq\overline{C}(n,\lambda_{\_}{n_{\_}0})$. To do so, we aim to show that the intersection between the curves $\overline{C}(n,\lambda)$ and $\overline{C}(n_{\_}0,\lambda)$ for any $n<n_{\_}0$ occur before $\lambda_{\_}{n_{\_}0}$. Combined with the properties of the curve $\overline{C}(n,\lambda)$ previously reported in Lemma 3, we can see in Fig. 12 that this is equivalent to what we are aiming to prove. Our goal is, therefore, summarized in proving that: $\frac{\overline{C}(n_{\_}0+1)-\overline{C}(n_{\_}0)}{A(n_{\_}0)-A(n_{\_}0+1)}\geq\frac{\overline{C}(n_{\_}0)-\overline{C}(n)}{A(n)-A(n_{\_}0)}$ for any $n<n_{\_}0$. From the first inequality of the results of Lemma 47, we can conclude that the series $\frac{\overline{C}(n_{\_}0)-\overline{C}(n)}{A(n)-A(n_{\_}0)}$ is increasing with $n$ for all $n\leq n_{\_}0-1$. Therefore, we have that for all $n<n_{\_}0$:

[TABLE]

Lastly, by using the fact that $\lambda(n)$ is increasing with $n$, we can conclude that: $\frac{\overline{C}(n_{\_}0+1)-\overline{C}(n_{\_}0)}{A(n_{\_}0)-A(n_{\_}0+1)}\geq\frac{\overline{C}(n_{\_}0)-\overline{C}(n_{\_}0-1)}{A(n_{\_}0-1)-A(n_{\_}0)}$. Combining this with the results of eq. (48), we can conclude our proof.

{strip}

Supplementary Material for the paper “The Age of Incorrect Information: A New Performance Metric for Status Updates”

Appendix C Proof of Lemma 1

Our proof is based on the well-known value iteration algorithm (VIA) [26]. By letting $V_{\_}t(.)$ be the value function at iteration $t$, the VIA consists of updating the value function as follows:

[TABLE]

Regardless of the initial value $V_{\_}0(S)$, it is well-known that the algorithm converges to the value function of the Bellman equation (14) [26] (i.e., $\lim_{\_}{t\to+\infty}V_{\_}{t}(S)=V(S)\>\>\forall S\in\mathbb{N}$). Consequently, to infer on the monotonicity of $V(S)$, it is sufficient to prove that $\forall S_{\_}2\geq S_{\_}1$:

[TABLE]

To proceed in that direction, and without loss of generality, we suppose that $V_{\_}0(S)=0\>\>\forall S\in\mathbb{N}$. Therefore, (50) holds for $t=0$. Next, we suppose that the condition in (50) is true up till $t>0$ and we examine if it holds for $t+1$. To do so, we examine the Right Hand Side (RHS) of (49) for both states $S_{\_}2$ and $S_{\_}1$. To that extent, we first take the case where $S_{\_}1\neq 0$ and we distinguish between the two possible transmission decisions $\psi$:

• •

$\psi=0$: In this case, the RHS is equal to $x=S_{\_}1+(p_{\_}R+(N-2)p_{\_}t)V_{\_}t(S_{\_}1+1)+p_{\_}tV_{\_}t(0)$ and $y=S_{\_}2+(p_{\_}R+(N-2)p_{\_}t)V_{\_}t(S_{\_}2+1)+p_{\_}tV_{\_}t(0)$ for $S_{\_}1$ and $S_{\_}2$ respectively. Baring in mind that $V_{\_}t(S_{\_}2)\geq V_{\_}t(S_{\_}1)$, we can easily see that $x\leq y$.

• •

$\psi=1$: In this case, the RHS is equal to $z=S_{\_}1+(p_{\_}Rp_{\_}f+(N-2)p_{\_}t+p_{\_}sp_{\_}t)V(S_{\_}1+1)+(p_{\_}Rp_{\_}s+p_{\_}fp_{\_}t)V(0)$ and $w=S_{\_}2+(p_{\_}Rp_{\_}f+(N-2)p_{\_}t+p_{\_}sp_{\_}t)V(S_{\_}2+1)+(p_{\_}Rp_{\_}s+p_{\_}fp_{\_}t)V(0)$ for $S_{\_}1$ and $S_{\_}2$ respectively. Taking into account that $V_{\_}t(S_{\_}2)\geq V_{\_}t(S_{\_}1)$, we can also verify that $z\leq w$.

Lastly, we know that if $x\leq y$ and $z\leq w$ then $\min(x,z)\leq\min(y,w)$. For the case where $S_{\_}1=0$, we can show that $x=z=p_{\_}RV_{\_}t(0)+(N-1)p_{\_}tV_{\_}t(1)$. After some algebraic manipulations, we can easily verify that the same above inequalities still holds. Consequently, we can assert that $V_{\_}{t+1}(S_{\_}2)\geq V_{\_}{t+1}(S_{\_}1)\>\>\forall t,S_{\_}1,S_{\_}2\in\mathbb{N}$. This concludes our inductive proof that shows that the value function $V(S)$ is increasing in $S\>\>\forall S\in\mathbb{N}$.

Appendix D Proof of Theorem 1

As we have previously stated, it is well-known that the optimal transmission policy can be obtained by solving the Bellman equation in (14). On top of that, we recall that the VIA, previously reported in the proof of Lemma 1, converges to the value function of the Bellman equation in (14). Consequently, we can deduce the optimal sequence of actions based on the value function at each time instant $t$ by reconsidering the VIA:

[TABLE]

To that extent, let us define $\Delta V_{\_}{t+1}(S)$ as the difference between the value functions if the transmitter sends a packet or remains idle for any state $S$. More specifically, we have that $\Delta V_{\_}{t+1}(S)=V^{1}_{\_}{t+1}(S)-V^{0}_{\_}{t+1}(S)$ where $V^{1}_{\_}{t+1}(S)$ and $V^{0}_{\_}{t+1}(S)$ are the value functions at time $t+1$ if $\psi=1$ and $\psi=0$ respectively. By obeying to the dynamics reported in Section III-B, we have:

[TABLE]

[TABLE]

The first thing we see is that when the state of the system is $S(t)=0$, both actions of remaining idle or transmitting leads to the same value function at time $t+1$. We can now tackle the case where $S(t)\neq 0$. To that extent, and as $V_{\_}t(S)$ is always increasing with $S$ (Lemma 1), we can assert that $(V_{\_}t(S+1)-V_{\_}t(0))\geq 0$. Based on this, we distinguish between the following cases:

D-1 $p_{\_}t<p_{\_}R$

In this scenario, we can see that $\Delta V_{\_}{t+1}(S)$ is always negative for any $S\neq 0$. Consequently, it is always optimal to transmit a packet when $S(t)\neq 0$. Combined with the fact that a transmission or remaining idle leads to the same value function when $S(t)=0$, we can conclude that the optimal policy is to either send updates at each time slot or send updates when the receiver is in an erroneous state (i.e., when $S(t)\neq 0$). To calculate the average cost in this case, we can see that in the case of an “always update” policy, the MDP can be modeled through a Discrete Time Markov Chain (DTMC) where:

• •

The states refer to the values of the penalty function $S(t)$.

• •

The dynamics of $S(t)\>\>\forall(S,t)$ coincide with those of $\psi(t)=1$ of Section III-B.

The DTMC mentioned above is reported in Fig. 13.

To find the average cost in this case, we first provide the following lemma.

Lemma 9**.**

The DTMC of the “always update” policy is irreducible and admits $\pi_{\_}k\>\>\forall k\in\mathbb{N}$ as its stationary distribution where:

[TABLE]

[TABLE]

with the constant $a$ being equal to $p_{\_}Rp_{\_}f+(N-2)p_{\_}t+p_{\_}sp_{\_}t$.

Proof:

It is sufficient to formulate the general balance equations at any state $k\geq 2$, which leads to $\pi_{\_}k=a\pi_{\_}{k-1}$. By proceeding with a forward induction, and knowing that $\pi_{\_}1=(N-1)p_{\_}t\pi_{\_}0$, the results of (55) can be found. Next, by taking into account the fundamental equality $\sum\limits_{\_}{k=0}^{+\infty}\pi_{\_}k=1$, we can find $\pi_{\_}0$ which concludes our proof. ∎

To find the average cost of the above DTMC, we first note that the cost incurred by being at state $S=k$ is nothing but the value $k$ of the state itself. Consequently, we have that $\overline{C}_{\_}{AU}=\sum\limits_{\_}{k=1}^{+\infty}k\pi_{\_}k$. By taking into account the above stationary distribution, and the following series equalities, the expression in (15) can be found.

[TABLE]

D-2 $p_{\_}t\geq p_{\_}R$

In this case, we can see that $\Delta V_{\_}{t+1}(S)$ is always positive. Combined with the fact that a transmission or remaining idle leads to the same value function when $S(t)=0$, we can conclude that the optimal policy is always to remain idle. The intuition behind this is that when $p_{\_}t\geq p_{\_}R$, any packet being transmitted about the information source has a high chance of becoming obsolete by the time it reaches the monitor. To calculate the average cost in the case where the transmitter is always idle, the MDP can be modeled through the DTMC reported in Fig. 14.

The analysis of the above DTMC is the same as the one of the previous case ($p_{\_}t<p_{\_}R$). More specifically, it is sufficient to substitute $a$ by $b$ where $b=p_{\_}R+(N-2)p_{\_}t$ in (15) to obtain the expression in (16).

Appendix E Proof of Proposition 1

The proof follows the same direction as that of Theorem 1. More precisely, the optimal transmission policy can be obtained by solving the Bellman equation formulated in (23). To that extent, we leverage the VIA to find the optimal stransmission sequence. In other words, and as it has been done before, we investigate $\Delta V_{\_}{t+1}(S)=V^{1}_{\_}{t+1}(S)-V^{0}_{\_}{t+1}(S)$ where $V^{1}_{\_}{t+1}(S)$ and $V^{0}_{\_}{t+1}(S)$ are the value functions at time $t+1$ if $\psi=1$ and $\psi=0$ respectively. By obeying to the dynamics reported in Section III-B, we have:

[TABLE]

[TABLE]

As $\lambda\geq 0$, we can conclude that the action of remaining idle is always optimal when $S=0$. As for the case where $S\neq 0$, we can see that $\Delta V_{\_}{t+1}(S)$ is the sum of a positive constant and a decreasing non-positive function. Consequently, we have that the optimal action is increasing with $S$ from $\psi^{*}=0$ to $\psi^{*}=1$. In other words, the difference $\Delta V_{\_}{t+1}(S)$ decreases with $S$, and at a certain point, the action of transmitting becomes more beneficial than remaining idle. Therefore, we can conclude that the optimal policy of the problem is of a threshold nature.

Appendix F Proof of Proposition 2

To proceed with the proof, we first formulate the general balance equation at state $1$ which leads to $\pi_{\_}1(n)=(N-1)p_{\_}t\pi_{\_}0(n)$. Afterward, we provide the general balance equations at states $k$, with $2\leq k\leq n$:

[TABLE]

By noting the results above, along with those on $\pi_{\_}1(n)$, and by carrying on with a forward induction, the results of (27) can be found. Next, we formulate the balance equations at states $k$, with $k\geq n+1$:

[TABLE]

By using the above results, and those of (27), and by proceeding with a forward induction, we can find the equations in (28). Lastly, we make use of the following fundamental equality:

[TABLE]

By replacing $\pi_{\_}k(n)$ with their values in (61) and by noting the following series results:

[TABLE]

[TABLE]

we can find $\pi_{\_}0(n)$ which concludes our proof.

Appendix G Proof of Proposition 30

To calculate the average cost of the threshold policy, we first note that the cost incurred by being at state $S=k$ is nothing but the value $k$ of the state itself. Moreover, the transmitter attempts to send a packet solely when $S\geq n$. Consequently, we have that $\overline{C}(n,\lambda)=\overline{C}(n)+\overline{C}_{\_}1(n,\lambda)$ where:

[TABLE]

[TABLE]

By replacing $\pi_{\_}k(n)$ with its value from Proposition 2, we have that:

[TABLE]

To further simplify the above expression, we first note that the series $\sum\limits_{\_}{k=1}^{n}kb^{k-1}$ is nothing but the derivative with respect to $b$ of the series $\sum\limits_{\_}{k=0}^{n}b^{k}=\frac{1-b^{n+1}}{1-b}$. Consequently, by deriving the expression in the right hand side, we have that:

[TABLE]

Next, we can address the second term of the expression in (66). To that extent, we proceed with a change of variables $k^{\prime}=k-n$. With that being done, and by noting the fact that $\sum\limits_{\_}{k^{\prime}=1}^{+\infty}k^{\prime}a^{k^{\prime}}=\frac{a}{(1-a)^{2}}$, the expression in (29) can be found. By pursuing the same series analysis, we can deduce the expression in (30) which concludes our proof.