Utility Maximizing Sequential Sensing Over a Finite Horizon

TL;DR

This paper formulates a finite-horizon resource management problem with uncertain binary states as a POMDP, deriving near-optimal sensing and exploitation strategies with practical applications in spectrum access and marketing.

Contribution

It introduces a low-complexity policy for sequential resource sensing and utilization, approximating the intractable optimal strategy in a finite-horizon POMDP setting.

Findings

The proposed policy achieves near-optimal performance in simulations.

Optimal strategies involve time-varying thresholds for each resource.

Applications include spectrum access and marketing strategies.

Abstract

We consider the problem of optimally utilizing $N$ resources, each in an unknown binary state. The state of each resource can be inferred from state-dependent noisy measurements. Depending on its state, utilizing a resource results in either a reward or a penalty per unit time. The objective is a sequential strategy governing the decision of sensing and exploitation at each time to maximize the expected utility (i.e., total reward minus total penalty and sensing cost) over a finite horizon $L$. We formulate the problem as a Partially Observable Markov Decision Process (POMDP) and show that the optimal strategy is based on two time-varying thresholds for each resource and an optimal selection rule for which resource to sense. Since a full characterization of the optimal strategy is generally intractable, we develop a low-complexity policy that is shown by simulations to offer near optimal performance. This problem finds applications in opportunistic spectrum access, marketing strategies and other sequential resource allocation problems.

Figures

Equations

$$\mathbf{s}\triangleq (s_1,\dots ,s_N{)}^T\in \mathbf{S}\equiv \{0,1{\}}^N,$$

$$\mathbf{\omega }[0]$$

$${\omega }_i[0]$$

$$H_0^{{\varphi }_k}:s_{{\varphi }_k}=0,o[k]\stackrel{i.i.d}{\sim }{f}_0^{{\varphi }_k}(o)$$

$$H_1^{{\varphi }_k}:s_{{\varphi }_k}=1,o[k]\stackrel{i.i.d}{\sim }{f}_1^{{\varphi }_k}(o)$$

$$\displaystyle\underset{\mbox{\boldmath$\tau$},\mbox{\boldmath$\delta$},\mbox{\boldmath$\phi$}}{\text{max.}}$$

$$\tilde{\tau }\le L-1.$$

$$R_i(s_i;{\delta }_i)=\{\begin{array}{ll}{r}_i& \text{mbox}if{\delta }_i=s_i=0\\ -{\rho }_i& \text{mbox}if{\delta }_i=0,s_i=1\\ 0& \text{mbox}if{\delta }_i=1\end{array}$$

$$\mathbf{\omega }[k+1]$$

$${\omega }_i[k+1]$$

$${\Pi }_i({\omega }_i,o,\varphi )$$

$$\displaystyle V(\mbox{\boldmath$\omega$},\mathcal{A}_{k},k)$$

$$\displaystyle\triangleq\max\limits_{\mathcal{D}\subseteq\mathcal{A}_{k}}\left\{(L-k)V_{d}(\mbox{\boldmath$\omega$},\mathcal{D})+\max\limits_{i\in\mathcal{A}_{k+1}\equiv\mathcal{A}_{k}\setminus\mathcal{D}}V_{t}^{i}(\mbox{\boldmath$\omega$},\mathcal{A}_{k+1},k)\right\}$$

$$V_d^i({\omega }_i)$$

$$V_t^i(\mathbf{\omega },\mathcal{A},k)$$

$$f_{1-{\omega }_i}^{(i)}(o)\triangleq {\omega }_i{f}_0^{(i)}(o)+(1-{\omega }_i)f_1^{(i)}(o)$$

$$V_d^i({\omega }_i)=\max \{(r_i+{\rho }_i){\omega }_i-{\rho }_i,0\}.$$

$${\delta }_i^{\ast }=u\left(\frac{{\rho }_i}{{\rho }_i+r_i}-{\omega }_i\right)$$

$$V(\mbox{\boldmath$\omega$},\mathcal{A}_{k},k)=\max\limits_{\mathcal{D}\subseteq\mathcal{A}_{k}}J(\mathcal{D})$$

$$\displaystyle\!\!J(\mathcal{D})\triangleq\left\{(L-k)V_{d}(\mbox{\boldmath$\omega$},\mathcal{D})+\!\!\max\limits_{i\in\mathcal{A}_{k}\!\setminus\!\mathcal{D}}V_{t}^{i}(\mbox{\boldmath$\omega$},\mathcal{A}_{k}\!\setminus\!\mathcal{D},k)\right\}$$

$${\mathcal{D}}^{\ast }=\underset{\mathcal{D}\subseteq {\mathcal{A}}_k}{\mathrm{argmax}}J(\mathcal{D}).$$

$${\tau }_i^{\ast }=\{\begin{array}{ll}k,& \text{mbox}fori\in {\mathcal{D}}^{\ast }\\ k^{\prime }>k& \text{mbox}fori\in {\mathcal{A}}_k\setminus {\mathcal{D}}^{\ast }.\end{array}$$

$$\phi^{*}_{k}=\operatorname*{arg\,max}_{i\in\mathcal{A}_{k+1}\equiv\mathcal{A}_{k}\setminus\mathcal{D}^{*}}V_{t}^{i}(\mbox{\boldmath$\omega$},\mathcal{A}_{k+1},k)$$

$${\delta }_i^{\ast }=\{\begin{array}{ll}1& \text{mbox}if{\omega }_i[k]\le {\nu }_1^i({\mathcal{A}}_k,k)\\ 0& \text{mbox}if{\omega }_i[k]\ge {\nu }_0^i({\mathcal{A}}_k,k).\end{array}$$

$$\!\!V(\mbox{\boldmath$\omega$},\mathcal{A}_{k},k)\!=\!\max\left\{\max\limits_{\{i\}\subseteq\mathcal{D}\subseteq\mathcal{A}_{k}}J(\mathcal{D}),\max_{\mathcal{D}\subseteq\mathcal{A}_{k}\setminus\{i\}}J(\mathcal{D})\right\}$$

$${\nu }_1^i[k]$$

$${\nu }_0^i[k]$$

$${\nu }_1^i[k]$$

$${\nu }_0^i[k]$$

$${\nu }_1^i[k]\le {\nu }_1^i(\mathcal{A},k^{\prime })$$

$${\nu }_0^i[k]\ge {\nu }_0^i(\mathcal{A},k^{\prime })$$

$$(L-k)V_d^i({\omega }_i)=V_t^i({\omega }_i,i,k).$$

$$V_t^i({\omega }_i,i,k)\le -c+(L-k-1)r_i{\omega }_i.$$

$$R_i(s_i;{\delta }_i)\triangleq \{\begin{array}{ll}{C}_{00}^i>0& \text{mbox}if{s}_i=0,{\delta }_i=0\\ C_{10}^i<0& \text{mbox}if{s}_i=0,{\delta }_i=1\\ C_{11}^i>0& \text{mbox}if{s}_i=1,{\delta }_i=1\\ C_{01}^i<0& \text{mbox}if{s}_i=1,{\delta }_i=0\end{array}$$

$$\underline{{\nu }_1^i}[k]$$

Full text

I Introduction

We consider the problem of optimally utilizing $N$ resources over a horizon of length $L$. The state of each resource is either [math] (“good”) or $1$ (“bad”) and is unknown a priori. Utilizing resource $i$ results in either a reward $r_{i}$ or a penalty $\rho_{i}$ per unit time, depending on its state. To infer the states of the resources, only one of them can be sensed at each given time, accruing a random measurement drawn from a distribution determined by the state of the chosen resource. We study the optimal sequential strategy governing the decision of sensing and exploitation at each time, that maximizes the expected utility (i.e., total reward minus total penalty) over the finite horizon. Since sensing reduces the available time for utilization, the essence of the problem is in balancing the overhead associated with sensing and the penalty of utilizing resources in a bad state. Due to the limited sensing capability (i.e., only one resource can be sensed at a given time), it is also crucial to choose judiciously which resource to sense at each time. This problem arises in cognitive radio systems where the resources are channels that can be either busy or idle.

I-A Main Results

We formulate the problem above as a partially observable Markov decision process (POMDP) with state given by the current time, the set of remaining resources for which a utilization decision has not been made, and the belief on the state of these resources (i.e., the posterior probability that a resource is in state [math]) in Section II. A policy of this POMDP consists of three components: a set of selection rules indicating which resources to sense at each time, a set of stopping rules, and the induced stopping times, indicating when to terminate the sensing phase of each resource, and a set of decision rules governing whether to exploit or discard a resource after sensing (see Section II-B).

In Section III we explicitly characterize the optimal decision rules and show that the optimal stopping rule is given by two time-varying thresholds that also depend on the resources with pending decision and, since characterizing the time-varying thresholds and the optimal resource selection rules is analytically intractable, we develop a low-complexity suboptimal policy (described in III-B). Based on insights gained from the monotone properties of the thresholds in the single-resource case, we develop a recursive algorithm for computing approximate thresholds. For the resource selection rule, we propose an index policy based on an interchange argument. This index rule prioritizes resources according to the integrated effect of their utilization reward $r_{i}$, belief values, and the quality of the sensing measurements (which determines the speed of the state inference). In Section IV, we showcase two applications of our framework: cognitive radio and marketing strategy. In Section V we present simulation results to corroborate our theoretical findings. The asymptotic optimality of the index selection rule is also discussed.

I-B Related Work

A majority of existing work on sequential sensing focuses on typical detection metrics: minimizing the expected detection delay subject to constraints on detection accuracy in terms of the probabilities of false alarm and miss detection. This body of work can be partitioned into two general categories: passive sequential hypothesis testing and active sequential hypothesis testing. The former was pioneered by Wald in 1947[1], which introduced the procedure known as Sequential Probability Ratio Test (SPRT). Under this formulation, the observation model under each hypothesis is predetermined, and a test only needs to determine when to stop taking observations and which hypothesis to declare at the time of stopping. The latter was pioneered by Chernoff in 1959[2]. Active hypothesis testing has a control aspect that allows the decision maker to choose the experiment to be conducted at each time. Different experiments generate observations from different distributions under each hypothesis. An active test thus includes a selection rule (i.e., which experiment to carry out at each given time) in addition to a stopping rule and a declaration rule. Following these seminal results, there has been an extensive body of work on both passive (see, for example, [3, 4, 5, 6, 7, 8] and references therein) and active (see [9, 10, 11, 12, 13, 14] and references therein) hypothesis testing.

There are a couple of studies on sequential sensing for anomaly detection under the objective of minimizing operational cost [15, 16] which can be considered as a utility function. Different from this paper, these studies either restrict admissible sensing strategies to those that declare the state of one process before starting sensing another process (i.e., no switching across processes) [15] or focus on asymptotically (when the horizon length approaches infinity) optimal policies.

In the Cognitive Radio literature our utility maximization problem can be seen as a parametric formulation of the energy efficiency metric maximization [17, 18]. In [17] the presence of PU communication is inferred via energy detection (known to be optimal when no prior information is available) and a single PU channel was considered. Multiple PU channels and the capability of switching between different channels, known as spectrum handoff, is investigated in [19] and an efficient convex optimization procedure is developed to solve for the optimal values of the sensing slot duration and the channel switching probability in order to minimize energy consumption while guaranteeing satisfying throughput. In [18] prior knowledge (the vector $\boldsymbol{\omega}$ in our model) over the state of different channels is considered, but the sequential decision process terminates when the SU decides to transmit over one single channel. The problem is formulated as DP but no heuristic is provided to tackle the combinatorial complexity and no further insight on the threshold structure of the decision is proposed. The threshold structure for the channel probing stopping rule that has been proved in [20] and that [18] refers to, also considers only one possible transmission and a constant data time scenario, i.e. the transmission time is not affected by the time spent in sensing the channels. More general utility maximization approaches for cognitive networks can be found in [21] and [22], which leverage the class of utility maximization functions for optimal congestion and contention control introduced in [23]. A censored truncated sequential spectrum sensing procedure is presented in [24]111A review on censoring and sequential procedures can be found in the references of [24], where different cognitive radios sense the same channel and decide whether to send their estimates to a fusion center that then performs the final decision over the presence of a PU. We instead consider a single cognitive radio that can sense different channels (but only one at a time), which could be seen as using only one sensor per channel, and therefore each sensor is sensing a different state. The limit of one sensing operation at a time, in our formulation, could be seen as a rigid censoring constraint, with the possibility of suspending the decision over a channel, while continuing to sense and/or exploit others, and then potentially reconsider whether to sense or exploit that channel after some time instants. A concatenation of SPRT is proposed as a low-complexity, asymptotically optimal policy in [25].

Additional relevant works can be found in [26, 27, 28, 29, 30, 31, 32, 33, 34, 35]. The vast majority considers a rigid separation between exploration and exploitation phase, while in our framework we enable a combination of the two over different resources, by accessing some channels and simultaneously sensing a different resource at each time. For the application projection of our work, we will consider spectrum sensing for cognitive radios and market strategy with online advertisement. Notice that in both these applications, several works have adopted a Multi-Armed Bandit (MAB) formulation (see for example [35, 36] for cognitive radios , [37, 38] for online advertising), whereas others (including this work) followed a Partially Observable Markov Decision Process (POMDP) framework [27, 39]. It is important to highlight the following difference: in the MAB formulation the utility obtained by the player that selects a certain "arm" (i.e. an action) is the only information that is used to optimize the future choices. The concept of POMDP, instead, can be used to cast a wider class of decision problems where at each time epoch a certain action needs to be designed, that provides indirect information on what strategy the player should use to harness the maximum utility. In other words a ’sensing’ action informs what the player should do. The MAB formulation is a special case of the POMDP in which the sensing action and the action that brings the utility to the player are the same thing. Therefore, the POMDP formulation allows in principle for a richer action space than a MAB problem and a POMDP cannot necessarily be mapped into a MAB problem.

The key difference between this work and the vast body of results on hypothesis testing is that the design objective in the problem studied here is the utility maximization that directly addresses the trade-off between exploration (i.e., detecting the state of each resource) and the time that remains for exploitation (of the resources, based on the information gathered during the sensing phase). Hypothesis testing problems, passive or active, are pure exploration problems. With respect to the works in [15]-[16], the problem considered in this paper allows switching across resources and focuses on a finite horizon, resulting in a full-blown POMDP that is significantly more difficult to tackle (for a discussion on the general complexity of POMDP see [40]). In [17]-[19] no prior information is available and all the channels are equal. Therefore, there is no ordering of the channels to take into account and no SPRT procedure to be optimized for the sensing (performed via energy detection with a deterministic sensing time), whereas our paper tackles both aspects. The difference of our model with respect to a concatenation of truncated SPRT (and therefore a standard truncated SPRT) [25] is highlighted at the end of Section II, considering the value function of the POMDP associated with our problem and in the comparison presented in the simulation results in Section V. It is useful to remark that, by considering a time-dependent utility after the decision, in our model the constraints on the detection metrics vary between channels and over time, whereas in the majority of other works the detection metrics constraint are typically constant over time (see [1] for analysis of truncated sequential hypothesis). To be more precise, our model can be seen as a Bayesian-risk formulation, where our utility terms can be seen as the Lagrangian multiplier of the constraints associated to the detection metrics, that change over time in light of the time-dependent utility function. It is clear that when $L$ approaches infinity, the strategy developed in this paper would not offer a significant advantage with respect to other simplest quasi-static strategies (as discussed in [41] for a single resource case) or the concatenation of SPRT proposed in [25]. However, for a finite time horizon, the proposed strategy offers a better performance, as discussed both analytically at the end of Section II and shown numerically in Section V. Even if the behavior of the decision thresholds can be intuitive, our approach to approximate the decision thresholds (Algorithm 1) and the bound on the expected sensing time (presented in Lemma 6 and used in Algorithm 2), that form our low-complexity heuristic, are completely novel and, in light of the time-dependence and the absence of a close form expression to map the detection metrics constraints into Lagrange multipliers for our formulation, cannot be mapped directly into previous results. The discussion on the asymptotic regret in Section III-C is reported for completeness, but the asymptotic analysis is rather straightforward and does not represent the main focus of this work.

II Problem Formulation

II-A Problem Statement

Assume an agent has $L$ instants of time available for the sensing and exploitation of a set of resources $\mathcal{N}=\{1,2,\dots,N\}$. The agent accrues a reward that is a function of an underlying state vector

[TABLE]

where the entries $s_{i}\in\{0,1\}$ and $\mathbf{S}$ is the set of all possible states. We consider the $s_{i}$ as indicators of good ([math]) or bad ($1$) state of a resource. For instance, the “idle” or “busy” state of a sub-band, where the decision maker has to explore/sense the channels and gets a reward for utilizing an idle channel and a penalty for utilizing a busy channel. We assume the states $s_{i}$ are mutually independent Bernoulli random variables with known prior probabilities given by:

[TABLE]

Let $\mathcal{A}_{k}$ denote the set of resources for which a final decision of utilizing or discarding has not been reached at time $k$ ($k=0,1,\ldots,L-1$). Clearly, $\mathcal{A}_{0}$ includes all $N$ resources. Our model then allows to access multiple resources (removed from $\mathcal{A}_{k}$) and to sense one resource ($\phi_{k}$) at each time instant $k$. Underneath the decision there is a sequential binary hypotheses testing problem where a sample $o[k]$ from the selected resource $\phi_{k}$ is collected and the conditional probability density functions of the observations are assumed to be known:

[TABLE]

We want to maximize a utility function that strikes the best trade-off between the need of acquiring information on the state $\mathbf{s}$ of the channels and the desire of exploiting good resources as early as possible. The decision maker needs to design: 1) a set of $N$ stopping rules for the stopping times \mbox{\boldmath\tau}\triangleq\{\tau_{i}:i=1,2,\ldots,N\}, one for each resource, indicating when a final decision of utilizing or discarding the resource can be made; 2) a set of $N$ decision rules \mbox{\boldmath\delta}\triangleq\{\delta_{i}\in\{0(\mbox{utilize}),1(\mbox{discard})\}:i=1,2,\ldots,N\}, one for each resource, indicating the final decision at the time $\tau_{i}$ of stopping; 3) a sequence of selection rules \mbox{\boldmath\phi}\triangleq\{\phi_{k}:k=0,1,\ldots,L-1\} indicating which resource to sense at time $k$ (if $\mathcal{A}_{k}$ is not empty). Let $\tau_{i}$ denote the time instant at which a final decision on whether to utilize or discard resource $i$ is made and $\tilde{\tau}=\max_{i}{\tau_{i}}$ be the total sensing time, all depending on the two sets of rules (\mbox{\boldmath\tau},\mbox{\boldmath\phi}). The decision maker’s actions are the solutions to the following optimization problem:

[TABLE]

where the objective function has two terms:

• •

$-c\tilde{\tau}$ represents an effective sensing cost, and $c$ is the sensing cost per unit of time;

• •

the function $R_{i}(s_{i};\delta_{i})$ is the utility per unit time for exploiting resource $i$:

[TABLE]

where $r_{i},\rho_{i}>0$ indicate, respectively, the reward and the penalty for utilizing a good resource and utilizing a bad resource.

We can immediately notice an asymmetry in the function corresponding to the exploitation reward in (7) since, if one decides the resource is in bad state (i.e. that $s_{i}=1$), the utility accrued is the same regardless of the real state of the channel.

Note that, since $c$ can be moved out of the expectation in (6), \mathds{E}\left[U^{*}(\mathbf{s},\mathcal{N},L,c)\right]=\underset{\tau,\delta,\phi}{\max}~{}\mathds{E}\left[U(\mathbf{s},\mathcal{N},L,c,\mbox{\boldmath\tau},\mbox{\boldmath\delta},\mbox{\boldmath\phi})\right] decreases monotonically with $c$, therefore the value of $c$ is typically limited by an effective cost (i.e. the energy required for the receiver to continue sensing a channel in a spectrum access problem) and is not chosen to further optimize the achievable expected utility.

II-B A POMDP Formulation

In this subsection, we will model the optimization in (6) as a Partially Observable Markov Decision Problem (POMDP), and use dynamic programming tools to describe the optimal \mbox{\boldmath\tau}^{*},\mbox{\boldmath\delta}^{*},\mbox{\boldmath\phi}^{*}. The state vector $\mathbf{s}$ is not directly observable, therefore the decision maker has to rely on her belief regarding the occupancy of the resources in order to make a decision. Let $\boldsymbol{\omega}[k]$ denote the belief vector at instant $k$ (i.e., the vector of posterior probabilities that a resource is in state [math] given all past observations). We can use Bayes rule to derive the belief update after a new observation and write

[TABLE]

The optimization problem in (6) can therefore be formulated as a POMDP where the belief vector $\boldsymbol{\omega}[k]$ represents the state for the decision maker and the state transitions equations are given by the belief update rule in (8). At each time instant $k$, based on the current belief \mbox{\boldmath\omega}[k], the decision maker first removes from $\mathcal{A}_{k}$ all those resources for whom a final decision of utilizing or discarding can be made (we refer to this set as $\mathcal{D}$), and then she chooses one of the remaining resources to sense at time $k$. The value function of the POMDP problem can be expressed as:

[TABLE]

where V_{d}(\mbox{\boldmath\omega},D) indicates the maximum expected reward given by the resources in $\mathcal{D}$ for each of the $(L-k)$ remaining instants and V^{i}_{t}(\mbox{\boldmath\omega},\mathcal{A},k) represents the value accrued for deciding to sense the resource $i$ at time $k$ (i.e. $\phi_{k}=i$). It is easy to see from the utility function (7) that $V_{d}$ has an additive structure, i.e. V_{d}(\mbox{\boldmath\omega},\mathcal{D})=\sum_{i\in\mathcal{D}}V_{d}^{i}(\omega_{i}). Mathematically the functions $V_{d}^{i}(\omega_{i})$ and $V_{t}^{i}(\omega_{i},\mathcal{A},k)$ can be expressed as follows:

[TABLE]

for $k=0,1,\dots,L-1$ where we need a final condition V(\mbox{\boldmath\omega},\mathcal{A},L)=0,\forall\mathcal{A},\mbox{\boldmath\omega}, to encode the constraint in (6). In (13) we have defined

[TABLE]

It is easy to see that

[TABLE]

Thus the optimal final decision $\delta^{*}_{i}$ is given by

[TABLE]

where $u(\cdot)$ is the unit step function. The value function V(\mbox{\boldmath\omega},\mathcal{A}_{k},k) can be seen as the result of the maximization of a set function under set constraints, that is:

[TABLE]

where

[TABLE]

and this formulation will be used in the remainder of this work to show the structure of the optimal policy. We would like to point out that the formulation of the POMDP in (11) gives no indication on the fact that the channel sensed at time $k$ should continue to be sensed at time $k+1$ or included in the set $\mathcal{D}$, i.e. a concatenation of independent truncated SPRT over each channel represents a suboptimal strategy for our problem and is optimal only for $L\rightarrow\infty$.

Remark 1

Our formulation can be modified to account for correlation between group of resources. If we keep the condition that it is possible to sense only one resource at each time, the belief update rule in (8) should be modified to update also the $\omega_{i}$’s of resources correlated with the sensed one. The subset of resources, that can be accessed at each time, should contain all the ones correlated with each other, i.e. if two resources are correlated, they should be added to $\mathcal{D}$ at the same time, due to the possibility of gaining knowledge on a resource for which a terminal decision has already been made. It follows that, for each group of correlated resources, we have to consider a sub multi-hypothesis problem for all the possible combinations of binary states, and find a similar approximation as the one presented in the next section for the decision regions in the plane of the belief vector over these resources (for details on the geometry of such structure see [42]). This in principle can be handled for small groups of correlated resources, but makes the problem even more complex.

III Opportunistic Sequential Sensing

III-A The Optimal Stopping Rule and Decision Rule

In this section we first describe the correspondence between the actions of our decision process as solution of (11) and the optimal rules (\mbox{\boldmath\tau}^{*},\mbox{\boldmath\phi}^{*}) introduced in II-A, that the decision maker needs to determine. Then we introduce a low-complexity policy to approximate the optimal action. Let us start by considering the solution of (17) at time $k$, i.e.:

[TABLE]

By definition of the set of optimal stopping times $\tau$ (see (6)), for each resource $i$, the optimal stopping time $\tau^{*}_{i}$ is:

[TABLE]

In light of the outer maximization over $\mathcal{D}$ in (11), the optimal selection rule for the decision maker will be:

[TABLE]

From (20) and (21) it emerges how the optimal selection rules \mbox{\boldmath\phi}^{*} and the optimal stopping times \mbox{\boldmath\tau}^{*} are coupled. We then introduce the following lemma:

Lemma 1

$\forall{\mathcal{A}_{k}}\subseteq\mathcal{N}$, $i,j\in\mathcal{A}_{k}$ and for $k=0,\dots,L-1$, both V^{i}_{t}(\mbox{\boldmath\omega},\mathcal{A}_{k+1},k), V(\mbox{\boldmath\omega},\mathcal{A}_{k},k) are convex functions of $\omega_{j}$.

Proof:

The proof is in Appendix -A. It is similar to the approach used in [43] to prove the convexity of the Q-functions.

Lemma 1 induces the structure of the optimal stopping rules, which describe the optimal stopping times $\tau^{*}_{i},~{}i=1,2,\ldots,N$ that we formally present in the following theorem:

Theorem 1

The optimal stopping time $\tau^{*}_{i},\forall i\in\mathcal{N}$ is described by two thresholds $(\nu_{1}^{i},\nu_{0}^{i})$ that depend on the remaining time and channels to be explored, i.e. ($\nu_{1}^{i}=\nu_{1}^{i}(\mathcal{A}_{k},k),\nu_{0}^{i}=\nu_{0}^{i}(\mathcal{A}_{k},k)$) such that at any $k$ the optimum action for the resource $i$ is

• •

*take a final decision over resource $i$ (i.e. $\tau_{i}=k$) if *

$\omega_{i}[k]\geq\nu_{0}^{i}(\mathcal{A}_{k},k)~{}\vee~{}\omega_{i}[k]\leq\nu_{1}^{i}(\mathcal{A}_{k},k)$**

• •

*postponing the decision on resource $i$ (i.e. $\tau_{i}>k$), if *

$\nu_{1}^{i}(\mathcal{A}_{k},k)<\omega_{i}[k]<\nu_{0}^{i}(\mathcal{A}_{k},k)$**

*Furthermore $\forall i,\mathcal{A}_{k},k$

$\nu_{1}^{i}(\mathcal{A}_{k},k)\leq\frac{\rho_{i}}{\rho_{i}+r_{i}}\leq\nu_{0}^{i}(\mathcal{A}_{k},k)$ and from (16) it follows that:*

[TABLE]

Proof:

Following the same approach we used to show the convexity of V(\mbox{\boldmath\omega},\mathcal{A}_{k},k) in Appendix -A, we rewrite (11) for an arbitrary resource $i$ as follows:

[TABLE]

where the function $J(\mathcal{D})$ has been defined in (18), and then we refer to the two terms of the outer maximization as $f_{1}=\max\limits_{\{i\}\subseteq\mathcal{D}\subseteq\mathcal{A}_{k}}J(\mathcal{D})$ and $f_{2}=\max_{\mathcal{D}\subseteq\mathcal{A}_{k}\setminus\{i\}}J(\mathcal{D})$. Note that $f_{1}$ corresponds to the maximum value from the actions that immediately decide on resource $i$, whereas $f_{2}$ indicates the maximum value from the actions that do not decide on resource $i$ at time $k$. The convexity of $f_{1}$ and $f_{2}$ is proven in Appendix -A. Clearly $f_{1}\geq f_{2}$ for $\omega_{i}=0$ and for $\omega_{i}=1$ since, when the state of the channel is known, is clearly preferable (or equivalent) to immediately decide on it, and also $f_{1}$ is a piece-wise linear function of $\omega_{i}$ with only two segments that intersect in $\omega_{i}=\frac{\rho_{i}}{\rho_{i}+r_{i}}$ (see definition of $V_{d}^{i}(\omega_{i})$ in (15)). Hence, there are only two possibilities:

• •

$f_{1}<f_{2}$ for $\nu_{1}^{i}(\mathcal{A}_{k},k)<\omega_{i}[k]<\nu_{0}^{i}(\mathcal{A}_{k},k)$ where

$0\leq\nu_{1}^{i}(\mathcal{A}_{k},k)\leq\frac{\rho_{i}}{\rho_{i}+r_{i}}\leq\nu_{0}^{i}(\mathcal{A}_{k},k)$;

• •

$f_{1}\geq f_{2}$ $\forall\omega_{i}\in[0,1]$.

In this second case, we will say the two functions do not intersect, i.e. there is no region where the decision maker should prefer to not include the resource in $\mathcal{D}$; therefore we set $\nu_{1}^{i}(\mathcal{A}_{k},k)=\nu_{0}^{i}(\mathcal{A}_{k},k)=\frac{\rho_{i}}{\rho_{i}+r_{i}}$ to indicate the decision maker should immediately decide on resource $i$ and this concludes the proof.

It is not entirely surprising that the problem we defined leads to an optimal policy with a two thresholds structure, since analogous policies have been found to be optimal for a general truncated SPRT [44] with a deadline constraint either deterministic [45] or stochastic [46] and in general many different sequential decisions schemes. However, as described in the Introduction, our formulation is different in light of the time dependence of the Bayesian term, i.e. the different utility that a decision over a certain resource can produce at different times.

Theorem 1 is a description of the optimal stopping rules but does not indicate how to compute the two thresholds nor how to select the resource $\phi^{*}_{k}$ to be sensed at time $k$. Due to the recursive nature of the function $V_{t}^{i}(\boldsymbol{\omega},\mathcal{A}_{k+1},k)$ in (13) and the dependence on the rest of the system, the exact computation of these optimal thresholds remains elusive. In order to provide a suboptimal strategy that is computationally manageable in Section III-B, let us look at the situation where we have only one resource and we have to choose between testing the resource or taking a final decision, either $\delta_{i}=1$ or $\delta_{i}=0$. We will refer to the two single-resource decision thresholds as $\nu_{1}^{i}[k]=\nu_{1}^{i}(i,k),~{}\nu_{0}^{i}[k]=\nu_{0}^{i}(i,k)$, where we introduce this short notation for convenience. We then introduce the following lemmas.

Lemma 2

$\forall i\in\mathcal{N},~{}k=0,1,\dots,L-2$* the two single-resource thresholds will monotonically contract, i.e.*

[TABLE]

Proof:

The intuition behind this lemma is that, as the decision deadline approaches, the urge to decide whether to utilize or discard the resource increases and the decision thresholds shrink. For a rigorous proof see Appendix -B.

Lemma 3

$\forall i\in\mathcal{N},~{}k=0,1,\dots,L-1,~{}\mathcal{A}^{\prime}\in 2^{\mathcal{N}-i}$, the following inequalities hold

[TABLE]

Proof:

The intuition behind this lemma is that adding resources to the state $\mathcal{A}$ produces a similar effect to removing time dedicated to each resource. Thus a similar intuition as for Lemma 2 applies. The full proof is in Appendix -C.

Lemma 4

$\forall k,k^{\prime}=0,\dots,L-1,~{}k^{\prime}\geq k,~{}\mathcal{A}\in 2^{\mathcal{N}},i\in\mathcal{A}$**

[TABLE]

Proof:

Follows from Lemmas 2-3 and setting $\mathcal{A}=\mathcal{A}^{\prime}+i$.

Lemma 4 gives us insight on the behavior of the thresholds $\left(\nu_{1}^{i}(\mathcal{A}_{k},k),\nu_{0}^{i}(\mathcal{A}_{k},k)\right)$ that motivates our heuristic strategy in Section III-B, which uses a single-channel thresholds approximation to replace the optimal thresholds. In the next section we will first discuss how to approximate the thresholds $\nu_{1}^{i}[k]$, $\nu_{0}^{i}[k]$ and then we will introduce a manageable strategy to sense the resources and decide which action to take based on the approximate decision thresholds.

III-B A Low Complexity Approximation Algorithm

For the rest of this subsection, we will use underline to indicate a lower bound and overline for upper bound of a given quantity or function, i.e. $\underline{a}\leq a\leq\overline{a}$. For our heuristic, as we will motivate in the following, we will use the bounds $\underline{\nu_{1}^{i}}[k]$ and $\overline{\nu_{0}^{i}}[k]$ that correspond to the single-resource decision thresholds, derived from the formulation previously introduced. In principle, the tighter are the bounds we can provide for the thresholds, the better our heuristic will perform. Let us start by finding simple bounds. The decision thresholds $(\nu_{1}^{i}[k],\nu_{0}^{i}[k])$ at time $k$ are the solutions with respect to $\omega_{i}$ of

[TABLE]

From the convexity of $V_{t}^{i}(\omega_{i},i,k)$ (see Lemma 1) and the values of the function for $\omega_{i}=0,1$ we know that (see Fig.2)

[TABLE]

Furthermore, $V_{d}^{i}(\omega_{i})$ is piece-wise linear with only two segments that intersect in $\omega_{i}=\frac{\rho_{i}}{\rho_{i}+r_{i}}$.

Remark 2

A more general formulation with $4$ different rewards/penalties for (7) would not alter the structure of the problem nor invalidate our results. Our definition has been motivated by the emphasis we want to put on the utilization of the resources, which we assume occurs only if the detected state is [math], but if one had defined

[TABLE]

then with some manipulations, one can find the strategy would have the same structure, where Fig.2 would be “rotated".

By intersecting the upper bound for $V_{t}^{i}(\omega_{i},i,k)$ in (31) with $(L-k)V_{d}^{i}(\omega_{i})$ it follows:

[TABLE]

where we consider $\nu_{1}^{i}[k]=\nu_{0}^{i}[k]=\frac{\rho_{i}}{\rho_{i}+r_{i}}$ if (30) has no solutions for $\omega_{i}\in[0,1]$. The bounds above can be very loose depending on the parameters of our problem and lead to a poor approximation. Potentially tighter upper and lower bounds than the ones directly obtainable from (33)-(34)-(35) can be found using a probabilistic approach to write the value function of our dynamic program. The general idea is to consider a worst and a best case scenario any time the belief over a certain resource has not crossed one of the two thresholds. This is possible since the belief update (for a given observation) and the value function are monotonically increasing functions of $\omega_{i}$ (details are reported in Appendix -D). Let us define the following functions for $s=0,1$

[TABLE]

where (36) is the complementary CDF of the updated belief, which is a random variable while (37)-(38) refer to the probability of not overcoming the belief thresholds.

We will then prove

Lemma 5

$\forall i\in\mathcal{N},k=0,1,\dots,L-1$* we can derive the following upper and lower bound for the function $V_{t}^{i}(\omega_{i},i,k)$*

[TABLE]

[TABLE]

*where we used the step function $u[k]=1$ for $k>0$ and $u[0]=0$ and the short notation $\bar{F}_{i}^{1}(\omega|\varphi,s)$ for $\bar{F}_{\omega_{i}[\ell+1]}^{1}(\omega|\varphi,s)$. 

Proof:

See Appendix -D. Notice that, since we evaluate these bounds at time $k$ in $\omega_{i}$, the bounds for the thresholds at time $\ell=k$ can be replaced with $\omega_{i}$ (same holds for the bound in (54) in Section IV-B).

Gathering these results, we can now present our thresholds approximation in Algorithm 1. Note that our approach is valid irrespective of the sensing model. In fact, in order to express in closed form the bounds in (39)-(40) one only needs to evaluate the functions $\bar{F}^{1}_{\omega_{i}[k+1]}(\omega|\varphi,s)$ in (36).

We highlight it is important to run the algorithm for decreasing value of $k$ in order to use potentially tighter bounds when we compute (39)-(40) for lower values of $k$. The last step in the algorithm is a direct consequence of Lemma 2. In Section V we will numerically illustrate how the tightness of the bounds from Algorithm 1 is highly dependent on the value of the parameters of our utility function and on the quality of the test (i.e the sensing $SNR$ $\zeta_{i}$ for our application).

Having introduced two methods to approximate the thresholds, we now introduce our low-complexity strategy. The pseudocode for our heuristic is presented in Algorithm 2. Interestingly, the distance between the two bounds can be used to find an on-line (not computable a priori) non trivial upper-bound for the cumulative utility loss of Algorithm 2.

Lemma 6

The expected sensing time for resource $i$, given we continue to sense it from time $k$ until we make a final decision at time $\tau_{i}$, can be upper-bounded as follows:

[TABLE]

where $\varsigma(x,y)\triangleq\log\left(\frac{x(1-y)}{(1-x)y}\right)$ and

[TABLE]

Proof:

See Appendix -E

The reason for immediately deciding on the resources whose belief has crossed one of the two single-channel thresholds follows from Lemma 4, which implies that at any time instant $k^{\prime}$ after $k$ the decision maker will always decide not to test resource $i$ adding $kV_{d}^{i}(\omega_{i})$ to the overall utility. Therefore postponing that decision after $k$ will give a lower or equal expected utility. In other words, if $V_{d}^{i}(\omega_{i})>0$ and we expect to select to exploit resource $i$ at some point, we will have a lower utility postponing that decision; if, instead, $V_{d}^{i}(\omega_{i})=0$ and we discard channel $i$ we accrue zero utility irrespective on when we take that action. Using larger thresholds than the optimal ones appears a legitimate choice for applications where the accuracy of the test has a larger impact on the performances than the time devoted to sensing.

We then approximate the optimal selection rule \mbox{\boldmath\phi}^{*} with the index $\frac{\omega_{i}[k]r_{i}}{\mathds{E}[\tau_{i}-k|k]}$ which is computed by using the upper-bound introduced in Lemma 6. The motivation for the index and its asymptotic optimality are discussed in Appendix -F.

III-C Asymptotic regret

Even if this work focuses on finite horizon, it is useful to briefly discuss the asymptotic regret. Let us define the regret $\Psi(L)$ as the cumulative loss over the entire horizon $L$

[TABLE]

where $L\sum_{i\in\mathcal{N}}\omega_{i}[0]r_{i}$ represents the maximum achievable expected utility by a genie that knows exactly the state of each resource. To find an asymptotically optimal strategy in term of regret per slot, i.e. a strategy that achieves $\lim_{L\rightarrow\infty}\frac{\Psi(L)}{L}=0$, one can simply consider a strategy that senses for a fixed amount of time growing with $\log L$, that would asymptotically achieve zero probability to declare the wrong status for each resource, and then directly derive the limit. This proposed static strategy would give a cumulative regret with order $O\left({\log{L}}\right)$. In our simulation, we will then consider, for the different proposed strategies, the growth rate of the regret with $\log L$, specifically $\lim_{L\rightarrow\infty}\frac{\Psi(L)}{\log{L}}$ to highlight the importance of the resource sorting in reducing this quantity. The static approach, however, is clearly not feasible in the regime of small $L$ and this motivates our study of the optimum policy structure and our heuristic for such scenarios.

IV Applications

IV-A Sequential Spectrum Sensing

We now use our sequential decision model to find a single-band spectrum sensing technique for cognitive radio systems. The decision maker is a secondary transmitter and each resource $i$ is associated with a “channel” to communicate with a secondary receiver. The term channel is used in a broad sense and it could represent a frequency band with certain bandwidth, a set of tones in an OFDM system or a collection of spreading codes in a Code Division Multiple Access (CDMA) system. We denote with $\gamma_{i}^{P,S}$ the two instantaneous Signal-to-Noise Ratios at the receiver for the primary and secondary communication respectively. $\xi_{i}$ represents the instantaneous $SNR$ from the secondary transmitter that could interfere with the primary communication and $\zeta_{i}$ indicates the instantaneous $SNR$ at the secondary transmitter for the primary transmitter, which is the information the secondary transmitter uses to detect the primary communication. We consider all the communication channels of interest as Rayleigh fading channels, therefore all the $SNR$’s are exponentially distributed, i.e.

[TABLE]

where $\bar{\eta}$ indicates the average value of $\eta$. The phase of the received signal is uniform. If the secondary transmitter decides to transmit over the $i$-th channel ($\delta_{i}=0$) and the Primary Transmitter is not transmitting ($s_{i}=0$) then it accrues a utility per time instant which is a function of ${\gamma}_{i}^{S}$. For our model, we decide to choose the outage rate ($R^{S}_{i,out}$) as the reward per time instant left ($r_{i}$), since it is reasonable to assume a slowly varying channel and that the instantaneous CSI (Channel Side Information) is not known at the receiver, therefore the secondary transmitter uses a constant data rate to transmit. We refer to a design parameter $P^{S}_{i,out}$ that indicates the probability that the system can be in outage, i.e. the probability that the secondary transmitter cannot successfully decode the transmitted symbols. Since we assume Rayleigh fading, we can express our reward per time instant left

[TABLE]

where the derivation can be found in [47]. If instead, the secondary transmitter decides to use the channel ($\delta_{i}=0$), interfering with the Primary Transmitter communication ($s_{i}=1$), then it receives a penalty equal to the loss in outage rate caused to the Primary User, considering its interference adds to the noise at the Primary Receiver. Let us in fact assume that the Primary Transmitter also transmits at a certain outage rate, given by its value of $P^{P}_{i,out}$. In presence of interference from the secondary user its effective transmission rate becomes

[TABLE]

where

[TABLE]

(for details see Appendix -G). The penalty $\rho_{i}$ is therefore defined as the loss in rate caused by the interference

[TABLE]

In our model, we assume that cross-channel interference is negligible, i.e. the only possible interference for the primary communication over channel $i$ is given by a secondary transmission over the same channel. For this work, we also assume that the decision maker has perfect knowledge of the averages $\bar{\gamma}_{i}^{P},\bar{\gamma}_{i}^{S},\bar{\xi}_{i},\bar{\zeta}_{i}$ and the designed $P_{i,out}^{P},P_{i,out}^{S}$ for all the $N$ resources. Our signal model considers standard AWGN, therefore when no primary transmission is present, the received baseband complex equivalent signal (only noise) samples are independent identically distributed $\mathcal{CN}(0,N_{0})$ (which stands for complex circularly symmetric zero mean normal Gaussian samples with variance $N_{0}$) whereas, when a primary transmission exists, since we have assumed Rayleigh fading, the received signal samples are $\mathcal{CN}(0,P_{i}+N_{0})$, where $P_{i}$ already accounts for the path-loss which we consider deterministic (see Section V-A for details on the PHY layer). It follows that a sufficient statistic for the detection problem is given by the absolute value squares of the samples of the received signal, which are $\chi^{2}_{2}$-distributed, i.e. exponential. We then consider $f_{0}^{i}(o),f_{1}^{i}(o)$ exponential with parameter $\theta_{0}^{i},\theta_{1}^{i}$ and the ratio $\frac{\theta_{1}^{i}}{\theta_{0}^{i}}=1+\bar{\zeta}_{i}$.

For this particular sensing model, the KL distances are known and the two terms $\hat{D}(f_{0}^{i}||f_{1}^{i}),\hat{D}(f_{1}^{i}||f_{0}^{i})$ can be computed directly from their definition in (42)-(43):

[TABLE]

We can then directly evaluate the upper-bound in (41) for $\mathds{E}[\tau_{i}-k|k]$. We can also give a closed form expression for the complementary $CDF$ $\bar{F}_{\omega_{i}[k+\ell]}^{\ell}(\omega|\varphi,s)$ in (36), that will be used for different results that we will present in the next section. For our specific sensing model, by simple algebra and use the belief update equation in (8) we can prove that:

[TABLE]

where $F_{\Gamma}^{\ell}(x)$ is the CDF of the sum of $\ell$ exponential random variables with unitary mean. The key step to derive (51)-(52) is noting that, conditioned on the state $s$ of the resource, we have that $\frac{o[k]}{\theta_{s}^{i}}\overset{i.i.d.}{\sim}Exp(1)$. Once we have a closed expression for $\bar{F}_{\omega_{i}[k+\ell]}^{\ell}(\omega|\varphi,s)$ for $s=0,1$ and expressions for $r_{i},\rho_{i}$ in (47)-(50) respectively, we can follow Algorithm 1 to find bounds on the thresholds and use Algorithm 2.

IV-B Marketing Strategy

Another potential application is in marketing. Consider a retail company trying to decide, from a set of $N$ potential products, which to buy from wholesales for a new season. The products are seasonal and lose their value at the end of the season. The length of the season is $L$ days. At the beginning of the season, the probability of a successful sale of product $i$ to a typical customer is $\omega_{i}[0]$. The company can influence the market by advertising product $i$ with a daily advertising cost of $c_{i}$. Assume that under daily advertising, the probability of successful sale evolves according to a Markov process with state space $[0,1]$. After $k$ ($k=1,2,\ldots$) days of advertising, assume that the probability of successful sale, $\omega_{i}[k]$, can be inferred, for example, from the number of clicks of the ads [39]. The decisions the company needs to make is when to stop advertising and if yes, whether to abandon product $i$ or buy from the whole sale and make it available for its customers (see Fig.3).

Suppose that after $k$ days of advertising, the company chooses to make product $i$ available. With stochastically static customer arrivals, the expected profit from product $i$ per day for the remaining $L-k$ days of the season is $\left[\omega_{i}[k]r_{i}-(1-\omega_{i}[k])\rho_{i}\right]$, where $r_{i}$ is the difference between the retail and the wholesale prices and $\rho_{i}$ is the daily holding cost of the unsold product. We assume that the company will stop advertising after deciding to buy in from wholesale and the probability of successful sale will stay at $\omega_{i}[k]$. Based on the well established equivalence of a POMDP with partially observable states (i.e., $s_{i}$ in the previous sections) to an MDP with belief (i.e., $\omega_{i}[k]$) as the fully observed states, we can see that the sequential decision problem studied in the previous sections apply to this marketing application assuming that the company advertises no more than one product each day and the Markovian transitions of $\omega_{i}[k]$ under daily advertising satisfies (8). In this case $o[k]$ is a latent random variable (not an observation) drawn from the mixture $\omega_{i}[k]f^{i}_{0}(o)+(1-\omega_{i}[k])f^{i}_{1}(o)$ (where $i$ is the advertised product at time $k$) such that

[TABLE]

From (53) we have that the structure of the value function applies to this model as well, where for this case $s_{i}$ is not an underlying binary state of the product, but is the outcome of the daily Bernoulli experiment in which the probability $\omega_{i}$ models the willingness of the customers to buy that product. Under this assumption the two probability distribution functions $f^{i}_{0}(o),f^{i}_{1}(o)$ are interpreted as a parameter model fitting for the behavior of $\omega_{i}$ from previous market data series relative to similar products. The motivation for assuming such model for the marketing application is that it is a relative simple model to fit but at the same time has a high flexibility given that $f_{0}$ and $f_{1}$ are general functions with no restriction other than being pdfs. Furthermore the behavior of the function in (53) is such that $\omega$ is more sensitive to changes when it takes intermediate values (i.e. around $\frac{1}{2}$) and tends to saturate at the edges, i.e. when it approaches either [math] or $1$. Indeed, it is reasonable to assume the advertisement can be more impactful if the customer’s uncertainty is higher, rather then when there is a clear indication of interest, in one way or the other, for a specific product. Also, this model takes into account the avalanche effect that a popular product tends to become more and more popular: if $\omega_{i}[k]$ increases, it is more likely $o[k]$ will be drawn from $f^{i}_{0}(o)$ (higher weight in the mixture) and this will increase $\omega_{i}[k]$ and so on. Therefore, since the action space and the expected utility correspond to the one described in our framework, our model applies. The bounds for the $V_{t}$ function need to be modified to account for the absence of a true underlying state. The upper bound becomes:

[TABLE]

but a generalized lower bound is not tight and therefore, we have only used (54) in our simulation. The proof for obtaining (54) is very similar to that given in Appendix D for the bounds in Lemma 5 and it is omitted for brevity. Notice that the bound for the expected sensing time in Lemma 6 does not hold in this case, due to the different nature of the random variables $o[k]$’s. Because of this, we cannot use the heuristic selection rule and, in the example shown in the simulation section, we have considered only the case in which the advertiser has a single product to decide on. The development of an analogous bound for the expected sensing time for this different model is left for future work.

V Simulation Results

We now present our simulation results for the two applications presented: cognitive radio and marketing.

V-A Cognitive Radio

We highlight that, for the case of a single resource, we can compute the optimal decision thresholds $(\nu_{1}^{i}[k],\nu_{0}^{i}[k])$ obtained by MonteCarlo evaluation of the integral in (13). In our simulations we will show the improvement in the utility we can get by using the decision threshold bounds obtained with our approach in Algorithm 1 instead of using the easy bounds in (33)-(34), that are obtained by convexity of the function $V_{t}$. For the cognitive radio case we have the following parameters: $\omega_{i}[0]=0.5,r_{i}=2,\rho_{i}=2,\bar{\zeta}_{i}=3$.

In Fig. 4(a) we can see how for $L\leq 10$ our thresholds approximation in Algorithm 1 can reach the same utility of the optimal decision thresholds, while for higher values of $L$ it acquires the same utility of the easy bounds obtainable from (33)-(34). The reason for this is that, when $L$ increases, our upper-bound for $V_{t}$ in (39), which is essentially a sum of upper-bounds, becomes looser. We also looked at the performance obtainable by constant decision thresholds (further referred to as CT strategy), i.e.

[TABLE]

$\forall k=0,1,\dots,L-1$ which always achieve worse utility than the others, being the looser approximation of the optimal thresholds. Notice that for higher value of $c$ and $\varrho$, in Fig.4(b), the optimal strategy is to not start sensing and simply accepting utility [math]. Our thresholds approximation (Algorithm 1) is able to capture this and follows the optimal strategy, while using the bounds in (33)-(34) or constant thresholds gives a negative utility. We can notice how the difference in utility between the optimal strategy and our approximation is higher for the marketing strategy application and developing new bounds for this case will be the object of future research. For the second experiment we analyze the case with multiple channels to be sensed for the cognitive radio application. We consider a heterogeneous network with $4$ Primary Transmitters located at the corners of a square with side $500m$. Our entire bandwidth goes from $800$ to $900MHz$ (see Fig. 5). A secondary transmitter wants to find opportunities to communicate with $N=20$ secondary receivers randomly spread around. The $20$ channels have a bandwidth of $5MHz$ and we assume they are equally divided among the $4$ Primary transmitter: observations are then collected at the Nyquist rate for the single channel, i.e. $2\mu s$. We assumed that the primary transmitter power is $10dBm$ and the height of the transmitter is $10m$, while for the secondary we chose power equal to $5dBm$ and a transmitter height of $3m$. The height of all the receivers is equal to $1m$.

We assume the secondary transmitter has an estimate for $\bar{\gamma}_{i}^{S}$ (the average $SNR$ at the secondary receiver from the secondary communication) and $\bar{\zeta}_{i}$ (the average $SNR$ at the secondary transmitter from the primary communication), considering a deterministic path loss propagation model. We used a deterministic two-ray model to predict the average $SNR$ received [48], i.e. the $P_{i}$ value introduced in Section IV-A $\left(\text{i.e.}~{}P_{i}=P_{t}G\frac{h_{t}^{2}h_{r}^{2}}{d_{i}^{4}}\right)$. The value of $G$ has been set to $2\cdot 10^{-4}$ to have average received $SNR$ ($\zeta_{i}$) at the secondary (in case of primary transmission) in the range of $5-10dB$, which is a reasonable range if we assume to only have thermal noise and no additional interference over the channel. The secondary transmitter does not know the position of the potential primary receiver but he estimates $\bar{\gamma}_{i}^{P}$ (the average $SNR$ at the primary receiver from the primary communication, without interference) and $\overline{\xi}_{i}$ (the average $SNR$ from the secondary transmitter to the primary receiver, that could interfere with an existing primary communication), by considering the closest point the primary receiver could be in the coverage area, i.e. the highest interference he could create (which does not necessarily correspond to the highest $\rho_{i}$). As explained in Section II-A, the value of $c$ is limited by the actual cost of testing. We want to study how the performances in terms of utility for different strategies (stopping rule and selection rule) change for different values of $\varrho$ and $c$. We compared the performances of Algorithm 2 with three possible alternatives:

A selection rule that follows an initial arbitrary order (indicated in our plot with $NS$= “No Sorting”), i.e. $\phi_{k}=\operatorname*{arg\,max}_{i\in\mathcal{A}_{k+1}}i$. 2. 2.

Constant decision thresholds (indicated in our plot with $CT$= “Constant Thresholds”), as in (55)-(56) $\forall i\in\mathcal{N}$ 3. 3.

both 1) and 2) (indicated with $CTNS$)

Notice that the NS procedure corresponds to conducting a sequence of concatenated truncated SPRT, which, as highlighted in Section II, represents a suboptimal strategy for our problem (even with optimal thresholds). We will study the performances in three different regimes of $L$. For low values of $L$ (generally $L\leq N$) the coupling of the problem becomes more relevant since there is no time for the decision maker to sense all the resources and the single-resource decision thresholds are a loose approximation of the actual optimal thresholds which are much tighter. In light of this, we add the following step in our Algorithm 2. After computing the quantities $\overline{\mathds{E}}\left[\tau_{i}-k|k\right]$ for $i\in\mathcal{A}_{k}$ from (41) and sort the resources according to our index in (44), we keep in $\mathcal{A}_{k+1}$ all the resources with higher index as long as the following condition is satisfied:

[TABLE]

and add to $\mathcal{D}$ the remaining ones. The motivation for this additional step is that, since we expect to not have time to sense all of them, we might start acquiring positive utility in expectation from some of them. In Fig.6 we indicate this additional removal with “AR”.

We can see that for $L\leq 10$ the CT modification of Algorithm 2 can achieve higher utility, by including the additional removal of resources, than the original Algorithm 2. For $L=20$ the importance of the additional removal becomes almost negligible for Algorithm 2, while it continues to have an impact on the CT modification. In this regime of L, we can see the importance of the selection rule by looking at the two strategies that do not sort the resources (dashed lines) and are clearly outperformed by the other strategies. In Fig. 7 we show the utility achieved by the $4$ strategies previously introduced: Alg 2, CT, NS, CTNS for moderate values of $L$ (i.e. $70\leq L\leq 100$). We can see how Algorithm 2 outperforms the other strategies and also that when we increase the cost $c$, the performances of CT, initially close to our complete heuristic, get worse and considering time-varying threshold becomes more important than sorting, i.e. the NS utility is higher than CT. The CTNS approach is always the worse. The same trend was observed for different values of $\varrho$.

Finally, we look at the regime for high $L$. Following the discussion in Section III-C, in Fig.8 we plot the growth rate of the regret $\frac{\Psi(L)}{\log{L}}$ for the $4$ different strategies. We can see this quantity is approximately constant, and the key factor to reduce this constant relies in the selection rule, other than the time-varying behavior of the decision thresholds. Similar trends were observed for different values of $c$ and $\varrho$ where both higher $c$ and $\varrho$ increase the regret $\Psi(L)$.

V-B Marketing

For the second application of online marketing strategy, as motivated in Section IV-B, we can only show the case for a single product. We assumed $f_{0}(o)\sim\mathcal{N}(0;1),f_{1}(o)\sim\mathcal{N}(0.75;1)$ and the following parameters: $\omega_{i}[0]=5,r_{i}=1,\rho_{i}=1$ (where $r_{i},\rho_{i},c$ are normalized over their unit measure, i.e. thousands of ). In Fig.[9](#S5.F9) we have plotted, similarly to what has been shown in Fig. [4](#S5.F4), the improvement in achievable utility obtained by our approximation for the bounds of V_{t}$ compared to the bounds in (33)-(34).

We highlight that for $L=5$ only the optimal decision thresholds can guarantee a positive utility, however our thresholds approximation algorithm achieves a loss which is only $6\%$ of the loss that one would obtain using the bounds in (33)-(34) and for $L=10$ our approximation algorithm utility is $40\%$ higher.

VI Conclusions

We proposed a new framework to design a sequence of tests over a finite time horizon and accrue a utility which is function of an unknown state vector and depends on the actions the decision maker takes at each time instant. We derived the structure of the optimal strategy and we proposed a heuristic to approximate the optimal decisions. Simulation results were reported to support our findings and have a physical understanding of the role played by the parameters in the utility accrued by the decision maker for two different applications: cognitive radio and marketing strategy.

-A Proof of Lemma 1

We first notice that \forall\mbox{\boldmath\omega},\mathcal{A}, V(\mbox{\boldmath\omega},\mathcal{A},L)=0 implies V^{j}_{t}(\mbox{\boldmath\omega},\mathcal{A},L-1)=-c,~{}\forall~{}j\in\mathcal{A} and therefore

[TABLE]

which is convex in any $\omega_{j},j\in\mathcal{A}$, since it is a positive sum of piece-wise linear functions (see (15)). We then prove the lemma by induction. We assume V(\mbox{\boldmath\omega},\mathcal{A},k+1) is convex in $\omega_{j}$ and we show V(\mbox{\boldmath\omega},\mathcal{A},k) is convex by showing all the possible functions V^{i}_{t}(\mbox{\boldmath\omega},\mathcal{A},k) are convex in $\omega_{j}$. We start by the case $j=i$. Without loss of generality, let us consider two $\omega$ say (\mbox{\boldmath\omega}^{1},\mbox{\boldmath\omega}^{2}) which differ only in the $j$-th entry (i.e $\omega_{\ell}^{1}=\omega_{\ell}^{2},\forall\ell\neq i$). We want to prove that for $0\leq\lambda\leq 1$ we have \lambda V^{i}_{t}(\mbox{\boldmath\omega}^{1},\mathcal{A},k)+(1-\lambda)V^{i}_{t}(\mbox{\boldmath\omega}^{2},\mathcal{A},k)\geq V^{i}_{t}(\lambda\mbox{\boldmath\omega}^{1}+(1-\lambda)\mbox{\boldmath\omega}^{2},\mathcal{A},k). From (13) we can write (we will use the short notation \mbox{\boldmath\omega}^{\ell,o}=\Pi(\mbox{\boldmath\omega}^{\ell},o,i) for $\ell=1,2$)

[TABLE]

where $(a)$ follows from the assumption that V(\mbox{\boldmath\omega},\mathcal{A},k+1) is convex in $\omega_{i}$ and $\mu=\frac{\lambda f^{i}_{1-\omega_{i}^{1}}(o)}{\lambda f^{i}_{1-\omega_{i}^{1}}(o)+(1-\lambda)f^{i}_{1-\omega_{i}^{2}}(o)}.$ Now, if we define \mbox{\boldmath\omega}^{3}=\lambda\mbox{\boldmath\omega}^{1}+(1-\lambda)\mbox{\boldmath\omega}^{2} we have

[TABLE]

since $f^{i}_{1-\omega_{i}}(o)$ is a linear affine function of $\omega_{i}$ and also

[TABLE]

which implies

[TABLE]

Therefore, by replacing (59),(60) in (58) we have

[TABLE]

and this proves the convexity of V^{i}_{t}(\mbox{\boldmath\omega},\mathcal{A},k) in $\omega_{i}$. The convexity of V^{i}_{t}(\mbox{\boldmath\omega},\mathcal{A},k) in $\omega_{j}$, $j\neq i$ can be proved by considering two points (\mbox{\boldmath\omega}^{1},\mbox{\boldmath\omega}^{2}) with the same $i$-th coordinate (i.e $\omega_{i}^{1}=\omega_{i}^{2}=\omega_{i}$) and following similar steps as before, where this time we have

[TABLE]

and we do not need to introduce $\mu$ to conclude our proof. To show the function V(\mbox{\boldmath\omega},\mathcal{A}_{k},k) is convex in $\omega_{j}$ we rewrite the maximization over $\mathcal{D}$ in (11) as follows:

[TABLE]

where the function $J(\mathcal{D})$ is defined in (18). Let us then call $f_{1}$ and $f_{2}$ the two maximizations inside (63) in the order they appear and let us omit the arguments for brevity. Now if we see $f_{1}$ and $f_{2}$ as functions of $\omega_{j}$ we have that both $f_{1}$ and $f_{2}$ are convex functions of $\omega_{j}$. In fact $\omega_{j}$ in $f_{1}$ appears only as argument of $V_{d}^{j}(\omega_{j})$ which is a piece-wise linear function of $\omega_{j}$ and therefore convex, whereas in $f_{2}$, $\omega_{j}$ is an argument of the second term of $J(\mathcal{D})$, which is the maximization over the index $i$ of the functions V_{t}^{i}(\mbox{\boldmath\omega},\mathcal{A}_{k+1},k) ($i,j\in\mathcal{A}_{k+1}$) that are all convex in $\omega_{j}$. Therefore the maximum is convex and we can conclude $f_{2}$ is convex. The convexity of V(\mbox{\boldmath\omega},\mathcal{A}_{k},k) follows from the fact it is the maximum of two convex functions.

-B Proof of Lemma 2

Let us first introduce the following lemma

Lemma 7

$\forall i\in\mathcal{N},\omega_{i}\in[0,1],k=0,\dots,L-2$:

[TABLE]

Proof:

We will prove once again by induction. First we show that (64) is true for $k=L-2$ (we use the short notation $\omega_{i}^{o}=\Pi_{i}(\omega_{i},o,i)$ and $dF_{i}^{o}=f^{i}_{1-\omega_{i}}(o)do$).

[TABLE]

where $(a)$ holds for the convexity of the function $V_{d}^{i}$ and $(b)$ holds for the martingale property of the prior belief update

[TABLE]

Then we show that if (64) holds for $k$, then it holds for $k-1$.

[TABLE]

where $(a)$ holds for the induction hypotheses, $(b)$ for the convexity of $V_{d}^{i}$ and $(c)$ for the martingale property of the belief update.

Now, to show the two thresholds respect (24)-(25) it is equivalent to prove the following statement

[TABLE]

and this can be proved since

[TABLE]

where $(a)$ holds by hypothesis and $(b)$ follows from Lemma 7, and this completes the proof.

-C Proof of Lemma 3

We start our proof by highlighting a property of the value function defined in (11) with the following lemma

Lemma 8

$\forall i\in\mathcal{N},\mathcal{A}^{\prime}\in 2^{\mathcal{N}-i},\omega_{i}\!\in\![0,1],k=0,\dots,L-1$**

[TABLE]

Proof:

Let us consider the function $J(\mathcal{D})$ defined in (18) refers to the value function when $\mathcal{A}_{k}=\mathcal{A}=\mathcal{A}^{\prime}+i$ and $J^{\prime}(\mathcal{D})$ refers to the value function when $\mathcal{A}_{k}=\mathcal{A}^{\prime}$. We will again use induction. Clearly (65) holds for $k=L-1$ where

[TABLE]

Then, assuming (65) holds for $k+1$, we prove it holds for $k$. Let us write (65) as

[TABLE]

and call $\tilde{\mathcal{D}}=\operatorname*{arg\,max}\limits_{\mathcal{D}\subseteq\mathcal{A}}J(\mathcal{D})$. There are three possible cases:

If $i\in\tilde{\mathcal{D}}$,

[TABLE] 2. 2.

If $i\notin\tilde{\mathcal{D}}$ and i=\operatorname*{arg\,max}\limits_{j\in(\mathcal{A})\setminus\tilde{\mathcal{D}}}V_{t}^{j}(\mbox{\boldmath\omega},\mathcal{A}\setminus\tilde{\mathcal{D}},k)

[TABLE]

where $(a)$ follows from the definition of our value function in (11) and $(b)$ holds by the induction hypothesis. 3. 3.

If $i\notin\tilde{\mathcal{D}}$ and i\neq j^{*}=\operatorname*{arg\,max}\limits_{j\in\mathcal{A}\setminus\tilde{\mathcal{D}}}V_{t}^{j}(\mbox{\boldmath\omega},\mathcal{A}\setminus\tilde{\mathcal{D}},k) the proof is similar to the previous case.

Proving (26)-(27) is equivalent to prove:

[TABLE]

To prove this, we call $\mathcal{D}^{*}=\operatorname*{arg\,max}\limits_{\mathcal{D}\subseteq\mathcal{A}^{\prime}}J^{\prime}(\mathcal{D})$ and use the chain of inequalities

[TABLE]

where $(a)$ holds by hypotheses, since if $(L-k)V_{d}^{i}(\omega_{i})\geq V_{t}^{i}(\omega_{i},i,k)$ then $V(\omega_{i},i,k)=(L-k)V_{d}^{i}(\omega_{i})$ and definition of $J^{\prime}(\mathcal{D})$, $(b)$ follows by Lemma 8 and $(c)$ by definition of value function in (11), and this completes the proof.

-D Proof of Lemma 5

Since we are focusing on a specific resource $i$ next, for brevity we drop the index $i$ and define the following events:

[TABLE]

We then write the function $V_{t}(\omega,k)$ as follows (we use the short notation $P_{0}(A)=P(A|s=0)$ and same for $P_{1}(A)$):

[TABLE]

To find bounds for $V_{t}(\omega,k)$ we will use:

an upper/lower bound for $P\left(\mathscr{C}_{k+1}^{\ell}\right)$ 2. 2.

an upper/lower bound for P_{0}\left(A_{\ell+1}\big{|}\mathscr{C}_{k+1}^{\ell}\right) 3. 3.

an upper/lower bound for P_{1}\left(A_{\ell+1}\big{|}\mathscr{C}_{k+1}^{\ell}\right)

First we write:

[TABLE]

We now find 2). 3) follows a similar approach and 1) will be otained using 2) and 3) (in our derivation with $P(\cdot|\varphi)$ we indicate we are conditioning on $\omega[\ell]=\varphi$).

[TABLE]

where the upper and lower bound start by taking out of the integral the maximum and minimum value of

$P_{0}(A_{\ell+1}|\omega[\ell]=t)$ and notice that

[TABLE]

Following similar steps for the event $B_{\ell+1}$ we can derive:

[TABLE]

Exactly the same bounds can be found for the case $s=1$, replacing the index of $P$ and the conditioned state on $F$ from [math] to $1$. For the case of $\ell=k$, which corresponds to the probabilities in the first term of the sum in (67), we can use the bounds in (70)-(71) by considering $\overline{\nu_{0}}[k]=\underline{\nu_{1}}[k]=\omega$. We can then use the recursion in (68) and the bounds previously found to derive:

[TABLE]

Once we upper and lower bound the probabilities for each term of the sum in (67), then the argument of the expectation is a monotonically increasing function of $\omega[\ell]$. Therefore, since we are conditioning on the event $\mathscr{C}_{k+1}^{\ell}$, the expectation will be lower or upper-bounded by choosing $\omega[\ell]=\underline{\nu_{1}}[\ell]$ or $\omega[\ell]=\overline{\nu_{0}}[\ell]$ respectively. Gathering these results and considering the convexity of $V_{t}^{i}(\omega_{i},i,k)$, we can write the upper and lower bound $\overline{V_{t}}(\omega,k),\underline{V_{t}}(\omega,k)$ as in (39)-(40) where we introduce the $\max$ to make sure our functions remain above $c$.

-E Proof of Lemma 6

Without loss of generality we can prove the lemma at $k=0$ and for general $k$ simply consider the time translation and the equivalent problem for $k^{\prime}=0,1,\dots,L-k-1$. For the remainder of the proof the index $k$ is used as the time index for the test of resource $i$ that starts at time [math]. We indicate with $\Lambda^{k}_{i}\triangleq\sum_{\ell=1}^{k}\log\left(\frac{f_{1}^{i}(o[\ell-1])}{f_{0}^{i}(o[\ell-1])}\right)$ the log-likelihood of the samples collected for resource $i$ up to time $k$. By considering the belief update in (8) and the optimal policy structure in Theorem 1, a final decision on resource $i$ can be made as soon as

[TABLE]

where $\Upsilon^{i}_{0,1}(\mathcal{A}_{k},k)\triangleq\varsigma(\omega_{i}[0],\nu_{0,1}^{i}(\mathcal{A}_{k},k))$, and we recall the resources in the set $\mathcal{A}_{k}$ are the resources for which a decision is still pending at time $k$. From (76) it follows that $\tau_{i}$

[TABLE]

is a stopping time. From (28)-(29) with $k^{\prime}=0$ we can deduce $\forall k=0,1,\dots,L-1,\forall\mathcal{A}_{k}\subseteq\mathcal{N}$

[TABLE]

Therefore

[TABLE]

is a stopping time always greater than $\tau_{i}$, i.e. $P(\overline{\tau}_{i}>\tau_{i})=1$. We then derive a bound on the expected value of $\overline{\tau}_{i}$ and this will also hold for $\tau_{i}$. We will use a similar technique as the one used for the thresholds approximation (see Proof of Lemma 5 in Appendix -D) to write

[TABLE]

where the bounds follow by $\overline{\tau_{i}}$ being the stopping time, therefore we know $\underline{\Upsilon_{0}^{i}}<\Lambda_{i}^{\overline{\tau_{i}}-1}<\overline{\Upsilon_{1}^{i}}$ and $\Lambda_{i}^{\overline{\tau_{i}}}>\overline{\Upsilon_{1}^{i}}$ or $\Lambda_{i}^{\overline{\tau_{i}}}<\underline{\Upsilon_{0}^{i}}$. By Wald’s identity we have

[TABLE]

and since $\mathds{E}_{0}\left[\Lambda_{i}^{1}\right]=-D(f_{0}^{i}||f_{1}^{i})$, $\mathds{E}_{1}\left[\Lambda_{i}^{1}\right]=D(f_{1}^{i}||f_{0}^{i})$ we can write

[TABLE]

and combining (84)-(85) (conditioned on the status of the resource) and evaluating the two bounds in (78)-(79) at any time $k\neq 0$ we can obtain the bound in (41).

-F Motivation for the index in our decision algorithm

The reason to approximate the optimal selection rule with the index $\frac{\omega_{i}[k]r_{i}}{\mathds{E}[\tau_{i}-k|k]}$ finds its motivation in the asymptotic utility growth. Let us consider $L$ large enough such that we can neglect the probability of taking a wrong decision over a certain resource, and we can assume the sensing time for each resource is not affected by having spent time sensing other resources before. We also further limit our strategy to sequentially sense each resource until a decision is made and then switch to a different resource. Then there is an optimal sorting for this strategy which maximizes the expected utility. To find such sorting, we use an interchange argument. Let us consider a pair of arbitrary resources, say $1$ and $2$, and show the sorting $1,2$ at time [math] with $L$ instants available is optimal if (we use the short notation $\mathds{E}[\tau_{i}]$ for $\mathds{E}[\tau_{i}-k|k]$ when $k=0$):

[TABLE]

where $(a)$ follows from the assumption the sensing time for the second resource is not affected by the time spent in sensing the previous one, which is a reasonable assumption for $L$ large enough. However, due to the time variant threshold and the finite horizon scenario, the expected sensing time strongly depends on the time $k$ the test starts, and is therefore affected by the sorting. Nevertheless, this index represents a good approximation that takes into account the identifiability of the resource and simulation results will corroborate our intuition.

-G Derivation of (48)

The instantaneous $SNR$ at the primary receiver in presence of interference from the secondary transmitter can be expressed as $\gamma_{i,\bar{\xi}_{i}}^{P}=\frac{\bar{\gamma}_{i}^{P}\left|h^{P}\right|^{2}}{1+\bar{\xi}_{i}\left|h^{S}\right|^{2}}$, where $h^{P},h^{S}$ are the two complex channel gains and for Rayleigh fading their absolute value squared is exponential with unitary mean. By following the same derivation as in [47] for the no interferer case, and assuming independence between $\left|h^{P}\right|^{2}$ and $\left|h^{S}\right|^{2}$ one can use the cdf of $Y=X_{1}-X_{2}$, where $X_{1}\sim Exp(\alpha_{1})$, $X_{2}\sim Exp(\alpha_{2})$:

[TABLE]

to derive (49), and consequently the effective rate for the primary receiver expressed by (48).