Second-best Beam-Alignment via Bayesian Multi-Armed Bandits

TL;DR

This paper introduces a Bayesian multi-armed bandit approach for mm-wave beam alignment, optimizing the balance between exploration and exploitation to improve alignment probability and throughput.

Contribution

It proposes a novel second-best preference policy based on Bayesian bandits, outperforming existing methods in beam alignment tasks.

Findings

Up to 30% improvement in alignment probability.

Superior performance over Thompson sampling and UCB methods.

Effective in analog beamforming simulations.

Abstract

Millimeter-wave (mm-wave) systems rely on narrow-beams to cope with the severe signal attenuation in the mm-wave frequency band. However, susceptibility to beam mis-alignment due to mobility or blockage requires the use of beam-alignment schemes, with huge cost in terms of overhead and use of system resources. In this paper, a beam-alignment scheme is proposed based on Bayesian multi-armed bandits, with the goal to maximize the alignment probability and the data-communication throughput. A Bayesian approach is proposed, by considering the state as a posterior distribution over angles of arrival (AoA) and of departure (AoD), given the history of feedback signaling and of beam pairs scanned by the base-station (BS) and the user-end (UE). A simplified sufficient statistics for optimal control is derived, in the form of preference of BS-UE beam pairs. By bounding a value function, the second-best preference policy is derived, which strikes an optimal balance between exploration and exploitation by selecting the beam pair with the current second-best preference. Through Monte-Carlo simulation with analog beamforming, the superior performance of the second-best preference policy is demonstrated in comparison to existing schemes based on first-best preference, linear Thompson sampling, and upper confidence bounds, with up to 7%, 10% and 30% improvements in alignment probability, respectively.

Equations

$${\mathbf{z}}_k=\sqrt{P_{\mathrm{tx},k}}{\mathbf{u}}_k^H{\mathbf{H}}_k{\mathbf{v}}_k\mathbf{s}+{\mathbf{w}}_k,\forall k\in \{0,1,\dots ,N-1\},$$

$${\mathbf{H}}_k={\alpha }_k{\mathbf{a}}_{\mathrm{r}}({\theta }_{\mathrm{r},k}){\mathbf{a}}_{\mathrm{t}}^H({\theta }_{\mathrm{t},k}),$$

$${\mathbf{a}}_{\mathrm{x}}({\theta }_{\mathrm{x}})=\frac{1}{\sqrt{M_{\mathrm{x}}}}{\left[1,e^{j\frac{2\pi d_{\mathrm{x}}}{\lambda }{\psi }_{\mathrm{x}}},\cdots ,e^{j(M_{\mathrm{x}}-1)\frac{2\pi d_{\mathrm{x}}}{\lambda }{\psi }_{\mathrm{x}}}\right]}^{\top },\mathrm{x}\in \{\mathrm{t},\mathrm{r}\},$$

$$\displaystyle\mathbf{z}_{k}\approx\sqrt{P_{\mathrm{tx,k}}}\alpha_{k}\Big{[}(\sqrt{G}-\sqrt{g}){\delta[A_{k},X]}{+}\sqrt{g}\Big{]}\mathbf{s}+\mathbf{w}_{k},$$

$$G=\underset{({\theta }_{\mathrm{r}},{\theta }_{\mathrm{t}})\in \mathcal{B}({\mathbf{u}}^{(i)},{\mathbf{v}}^{(i)})}{\min }|{\mathbf{a}}_{\mathrm{r}}({\theta }_{\mathrm{r}}{)}^H{\mathbf{u}}^{(i)}{|}^2|{\mathbf{a}}_{\mathrm{t}}^H({\theta }_{\mathrm{t}}){\mathbf{v}}^{(i)}{|}^2,\forall i,$$

$$g=\underset{({\theta }_{\mathrm{r}},{\theta }_{\mathrm{t}})\notin \mathcal{B}({\mathbf{u}}^{(i)},{\mathbf{v}}^{(i)})}{\max }|{\mathbf{a}}_{\mathrm{r}}({\theta }_{\mathrm{r}}{)}^H{\mathbf{u}}^{(i)}{|}^2|{\mathbf{a}}_{\mathrm{t}}^H({\theta }_{\mathrm{t}}){\mathbf{v}}^{(i)}{|}^2,\forall i.$$

$$Y_k=\frac{|{\mathbf{s}}^H{\mathbf{z}}_k{|}^2}{\Vert \mathbf{s}{\Vert }^2{N}_0{W}_{\mathrm{tot}}(1+\Lambda g)},$$

$$f(Y_k=y|X=x;A_k=a_{\mathrm{s}})={\left[\nu e^{-\nu y}\right]}^{\delta [a_{\mathrm{s}},x]}[e^{-y}{]}^{1-\delta [a_{\mathrm{s}},x]},$$

$$\nu \triangleq \frac{1+g\Lambda }{1+G\Lambda }.$$

$$\bar{R}(A_{\mathrm{d}},P_{\mathrm{d}},R_{\mathrm{d}}|b_0,{\mathcal{H}}_L)$$

$$\displaystyle\qquad\triangleq\frac{T_{\mathrm{fr}}-LT_{\mathrm{s}}}{T_{\mathrm{fr}}}\ \mathbb{P}\left(X=A_{\mathrm{d}}\Bigl{|}b_{0},\mathcal{H}_{L}\right)\hat{R}(R_{\mathrm{d}},P_{\mathrm{d}}),$$

$$\hat{R}(R_{\mathrm{d}},P_{\mathrm{d}})\triangleq R_{\mathrm{d}}\mathbb{P}\left[R_{\mathrm{d}}\le W_{\mathrm{tot}}{\log }_2\left(1+\frac{|{\alpha }_k{|}^2{P}_{\mathrm{d}}G}{N_0{W}_{\mathrm{tot}}}\right)\right].$$

$$\text{P0: }\underset{\mu }{\max }{\mathbb{E}}_{\mu }\left[\bar{R}(A_{\mathrm{d}},P_{\mathrm{d}},R_{\mathrm{d}}|b_0,{\mathcal{H}}_L)|b_0\right],$$

$$\displaystyle\max_{\mu}\mathbb{E}_{\mu}\left[\mathbb{P}\left(X=A_{\mathrm{d}}\Bigl{|}b_{0},\mathcal{H}_{L}\right)\Biggr{|}b_{0}\right]$$

$$\times \frac{T_{\mathrm{fr}}-L{T}_{\mathrm{s}}}{T_{\mathrm{fr}}}\underset{R_d\ge 0,0\le P_d\le P_{\max }}{\max }\hat{R}(R_{\mathrm{d}},P_{\mathrm{d}}),$$

$$b_{k+1}[x]=\frac{\exp \{m_{k+1}[x]\}}{{\sum }_{l\in \mathcal{I}}\exp \{m_{k+1}[l]\}},$$

$$m_{k+1}[x]=m_k[x]+J(Y_k)\delta [A_k,x],\forall x\in \mathcal{I},$$

$$J(y)\triangleq (1-\nu )y+\ln \nu .$$

$$b_{k+1}[x]\stackrel{(a)}{=}\mathbb{P}(X=x|{\mathcal{H}}_{k+1})$$

$$\stackrel{(b)}{\propto }f(Y_k=y|X=x,A_k=a_{\mathrm{s}},{\mathcal{H}}_k)\mathbb{P}(X=x|A_k=a_{\mathrm{s}},{\mathcal{H}}_k)$$

$$\stackrel{(c)}{=}f(Y_k=y|X=x,A_k=a_{\mathrm{s}})b_k[x]$$

$$\stackrel{(d)}{=}{\left[\nu \exp \left\{-\nu y\right\}\right]}^{\delta [a_{\mathrm{s}},x]}[\exp \{-y\}{]}^{1-\delta [a_{\mathrm{s}},x]}{b}_k[x]$$

$$\stackrel{(e)}{=}\exp \left\{-y+J(y)\delta [a_{\mathrm{s}},x]\right\}{b}_k[x],$$

$$b_{k+1}[x]$$

$$=\frac{\exp \{m_{k+1}[x]\}}{{\sum }_{l\in \mathcal{I}}\exp \{m_{k+1}[l]\}},$$

$$f(y|\mathbf{m},a_{\mathrm{s}})\triangleq \sum _{x\in \mathcal{I}}f(Y_k=y|X=x,A_k=a_{\mathrm{s}})b_k[x]$$

$$=\frac{\exp \{m[a_{\mathrm{s}}]\}}{{\sum }_{l\in \mathcal{I}}\exp \{m[l]\}}\nu e^{-\nu y}+\left[1-\frac{\exp \{m[a_{\mathrm{s}}]\}}{{\sum }_{l\in \mathcal{I}}\exp \{m[l]\}}\right]e^{-y},$$

$${\mathbf{m}}_{k+1}=\mathbf{m}+J(y)\mathbf{\delta }[a_{\mathrm{s}}],$$

$$r_k(\mathbf{m},a)=\{\begin{array}{ll}0,& k\in {\mathcal{T}}_{\mathrm{BA}},\\ \frac{\exp \{m[a]\}}{{\sum }_{l\in \mathcal{I}}\exp \{m[l]\}},& k=L.\end{array}$$

$$V_k^{\ast }(\mathbf{m})$$

$$q_L(\mathbf{m},A_{\mathrm{d}})=r_L(\mathbf{m},A_{\mathrm{d}})=\frac{\exp \{m[A_{\mathrm{d}}]\}}{{\sum }_{l\in \mathcal{I}}\exp \{m[l]\}},$$

$$q_k(\mathbf{m},a_{\mathrm{s}})={\int }_{{\mathbb{R}}^{|\mathcal{I}|}}{V}_{k+1}^{\ast }({\mathbf{m}}^{\prime })f({\mathbf{m}}_{k+1}={\mathbf{m}}^{\prime }|{\mathbf{m}}_k=\mathbf{m},A_k=a_{\mathrm{s}})\mathrm{d}{\mathbf{m}}^{\prime }$$

$$={\int }_0^{\infty }{V}_{k+1}^{\ast }(\mathbf{m}+J(y)\mathbf{\delta }[a_{\mathrm{s}}])f(y|\mathbf{m},a_{\mathrm{s}})\mathrm{d}y.$$

$$V_L^{\ast }(\mathbf{m})=\underset{A_{\mathrm{d}}\in \mathcal{I}}{\max }{q}_L(\mathbf{m},A_{\mathrm{d}})=\frac{\exp \{m[{A_{\mathrm{d}}}^{\ast }]\}}{{\sum }_{l\in \mathcal{I}}\exp \{m[l]\}}.$$

$$V_k^{\ast }(\mathbf{m})$$

Full text

I Introduction

Millimeter-wave (mm-wave) technology has emerged as a promising solution to meet the demands of future communication systems supporting high capacity and mobility, thanks to abundant bandwidth availability [1]. However, high isotropic path loss and sensitivity to blockages pose challenges in the design of these systems [2]. To overcome the severe signal attention, mm-wave systems leverage narrow-beam communications, by using large antenna arrays at base stations (BSs) and user-ends (UEs). However, narrow beams are highly susceptible to mis-alignment due to mobility and blockage, hence they require utilization of beam-alignment schemes, which may cause huge overhead.

Therefore, the design of beam-alignment schemes with minimal overhead is of paramount importance, and has been a subject of intense research. One of the earliest yet most popular schemes is exhaustive search [3], which scans sequentially through all possible BS-UE beam pairs and selects the one with maximum signal power for data communications. To reduce the delay of exhaustive search, iterative search is proposed in [4], where scanning is first performed using wider beams, followed by refinement using narrow beams. In the aforementioned heuristic schemes, the optimal design is not considered. To address this challenge, in our previous papers [5, 6, 7, 8], we considered the optimal design of interactive beam-alignment protocols that utilize 1-bit feedback from UEs. In [5, 6], we design a throughput-optimal beam-alignment scheme for a single UE and two UEs, respectively, and we prove the optimality of a bisection search; in [7], we optimize the trade-off between data communication and beam-sweeping in a mobile scenario where the BS widens its beam to mitigate the uncertainty on the UE position; in [8], we incorporate the energy cost of beam-alignment, and prove the optimality of a fractional search method. In our aforementioned papers [5, 6, 7, 8], the optimal design is carried out under the restrictive assumption of error-free single-bit feedback. However, this assumption may not hold in the presence of significant side-lobe gain and/or low signal-to-noise ratio (SNR).

The case of erroneous or noisy feedback is considered in recent work [9, 10], and our work [11]. A coded beam-alignment scheme is proposed in [11] to correct these errors, but with no consideration of feedback to improve beam-selection. A multi-armed bandit (MAB) formulation based on upper confidence bound (UCB) is proposed in [9], by selecting the beam based on the empirical SNR distribution. A hierarchical beam-alignment scheme based on posterior matching is proposed in [10]: therein, a first-best policy is formulated, which selects the most likely beam pair based on the posterior distribution on the AoA-AoD pair. However, as we will see numerically, both UCB and first-best policies are prone to errors due to under-exploration of the beam space.

In this paper, we propose a beam-alignment design with the goal to maximize the alignment probability and the average throughput during the data communication phase. We pose the problem as a Markov decision process (MDP), where the beam pair is chosen based upon the belief over the AoA-AoD pair, given the history of scanned beams and the received signal power. We identify a simplified sufficient statistic in the form of preference of the AoA-AoD beam pairs. We derive lower and upper bounds to the value function, based on which we propose a heuristic policy which selects the beam pair with the second-best preference. We show numerically that this policy strikes a favorable trade-off between exploration and exploitation: instead of greedily choosing the beam corresponding to the most likely AoA-AoD pair (first-best [10]), it chooses the second most likely one, leading to better exploration; at the same time, it avoids wasting precious resources to scan unlikely beam pairs, leading to better exploitation than other MAB techniques, such as linear Thompson sampling (LTS) [12] and UCB [9]. The proposed second-best scheme is shown to outperform first-best [10], LTS-based [12] and UCB-based [9] schemes by up to 7%, 10% and 30% in alignment probability, respectively.

The rest of the paper is organized as follows. In Sec. II, we present the system model. In Sec. III, we formulate the problem and our proposed solution strategy. In Sec. IV, we present numerical results, followed by final remarks in Sec. V.

II System Model

We consider a downlink scenario with one BS and one UE, as depicted in Fig. 1. Time is divided into frames of duration $T_{\mathrm{fr}}{=}T_{\mathrm{s}}N$, each with $N$ slots of duration $T_{\mathrm{s}}$. The frame is partitioned into two phases: a beam-alignment phase of duration $LT_{\mathrm{s}}$ ($L{<}N$ slots), followed by a downlink data communication phase, of duration $(N{-}L)T_{\mathrm{s}}$. Each beam-alignment slot is further partitioned into a pilot transmission phase, of duration $T_{\mathrm{pt}}$, followed by a feedback phase, of duration $T_{\mathrm{fb}}$, with $T_{\mathrm{s}}{=}T_{\mathrm{pt}}{+}T_{\mathrm{fb}}$. These are detailed next.

The BS and UE are equipped with uniform linear arrays (ULAs) with $M_{\mathrm{t}}$ and $M_{\mathrm{r}}$ antenna elements, respectively, and use analog beamforming. The signal received at the UE is

[TABLE]

where $P_{\mathrm{tx,k}}$ is the average transmit power of the BS; $\mathbf{s}\in\mathbb{C}^{S}$ is the transmitted signal with $S$ symbols with $\operatorname{\mathbb{E}}[\lVert\mathbf{s}\rVert_{2}^{2}]={S}$; $\mathbf{H}_{k}{\in}\mathbb{C}^{M_{\mathrm{r}}\times M_{\mathrm{t}}}$ is the channel matrix; $\mathbf{v}_{k}{\in}\mathbb{C}^{M_{\mathrm{t}}}$ is the BS beamforming vector with $\|\mathbf{v}_{k}\|_{2}^{2}=1$; $\mathbf{u}_{k}{\in}\mathbb{C}^{M_{\mathrm{r}}}$ is the UE combining vector with $\|\mathbf{u}_{k}\|_{2}^{2}=1$; $\mathbf{w}_{k}{\sim}\mathcal{CN}(\boldsymbol{0},N_{0}W_{\mathrm{tot}}\mathbf{I})$ is additive white Gaussian noise (AWGN), with one-sided power spectral density $N_{0}$ and system bandwidth $W_{\mathrm{tot}}$.

Channel Model: We use the extended Saleh-Valenzuela geometric model with a single-cluster [14], as adopted in several previous works (e.g., see [15, 16, 8]). In fact, typical mm-wave channels have been shown to exhibit one dominant cluster containing most of the signal energy [17]. The single-cluster channel is modeled as

[TABLE]

where $\theta_{k}{\triangleq}(\theta_{\mathrm{r},k},\theta_{\mathrm{t},k}){\in}[-\frac{\pi}{2},\frac{\pi}{2}]^{2}$ is the angle of arrival (AoA) and angle of departure (AoD) pair associated to the dominant cluster, with complex fading gain $\alpha_{k}$; $\mathbf{a}_{\mathrm{r}}$ and $\mathbf{a}_{\mathrm{t}}$ are the UE and BS array response vectors, respectively, defined as

[TABLE]

where $\psi_{\mathrm{x}}{=}\sin\theta_{\mathrm{x}}$, $d_{\mathrm{x}}$ is the antenna spacing, $\lambda{=}c/f_{c}$ is the wavelength at carrier frequency $f_{c}$, $c$ denotes the speed of light. We assume that during the duration of one frame $T_{\mathrm{fr}}$, $\theta_{k}$ remains unchanged, $\theta_{k}{=}\theta$, and $\alpha_{k}$ are i.i.d. Reyleigh fading in each slot with distribution $\alpha_{k}{\sim}\mathcal{CN}(0,\ell(d)^{-1})$, where $\ell(d)$ is the path loss at distance $d$ from the BS. In fact, the AoA-AoD pair change much slower than the channel gain [18].

Codebook structure: In slot $k$, the BS uses the beamforming vector $\mathbf{v}_{k}{\in}\mathcal{V}$ and the UE uses the combining vector $\mathbf{u}_{k}{\in}\mathcal{U}$, from the codebooks $\mathcal{V}$ and $\mathcal{U}$, respectively. We assume a sectored model [8], in which the AoA and AoD spaces are partitioned into sectors of equal beamwidth (as shown in Fig. 1 for the case of four sectors, this model approximates well analog beamforming). Accordingly, let $\mathcal{B}_{\mathrm{r}}(\mathbf{u}){\subseteq}[-\frac{\pi}{2},\frac{\pi}{2}]$ and $\mathcal{B}_{\mathrm{t}}(\mathbf{v}){\subseteq}[-\frac{\pi}{2},\frac{\pi}{2}]$ denote the AoA and AoD supports of the UE combiner and BS beamformer vectors $\mathbf{u}{\in}\mathcal{U}$ and $\mathbf{v}{\in}\mathcal{V}$, respectively, with equal beamwidth $|\mathcal{B}_{\mathrm{r}}(\mathbf{u})|{=}\frac{\pi}{|\mathcal{U}|},{\forall}\mathbf{u}{\in}\mathcal{U}$ and $|\mathcal{B}_{\mathrm{t}}(\mathbf{v})|{=}\frac{\pi}{|\mathcal{V}|},{\forall}\mathbf{v}{\in}\mathcal{V}$, where $|\mathcal{B}|$ denotes the measure $|\mathcal{B}|{\triangleq}\int_{\mathcal{B}}\mathrm{d}x$. We define $\mathcal{B}(\mathbf{u},\mathbf{v}){\triangleq}\mathcal{B}_{\mathrm{r}}(\mathbf{u}){\times}\mathcal{B}_{\mathrm{t}}(\mathbf{v})$ as the joint AoA-AoD support of $(\mathbf{u},\mathbf{v})$. We assume that the angular supports are mutually orthogonal and form a partition of the entire AoA-AoD space $[-\frac{\pi}{2},\frac{\pi}{2}]^{2}$, i.e., $\mathcal{B}(\mathbf{u},\mathbf{v}){\cap}\mathcal{B}(\tilde{\mathbf{u}},\tilde{\mathbf{v}}){=}\emptyset,\forall(\mathbf{u},\mathbf{v}){\neq}(\tilde{\mathbf{u}},\tilde{\mathbf{v}})$ and $\cup_{\mathbf{u}\in\mathcal{U}}\mathcal{B}_{\mathrm{r}}(\mathbf{u}){=}\cup_{\mathbf{v}\in\mathcal{V}}\mathcal{B}_{\mathrm{t}}(\mathbf{v}){=}[-\frac{\pi}{2},\frac{\pi}{2}]$. Let $(\mathbf{u}^{(i)},\mathbf{v}^{(i)}),i\in\mathcal{I}{\triangleq}\{1,2,\ldots,|\mathcal{U}||\mathcal{V}|\}$ be any ordering of combining and beamforming vectors, and $\mathcal{B}^{(i)}\triangleq\mathcal{B}(\mathbf{u}^{(i)},{\mathbf{v}}^{(i)})$ be their support. Let $A_{k}{\in}\mathcal{I}$ be the beam index of the combining and beamforming vectors scanned in slot $k$, so that $(\mathbf{u}_{k},\mathbf{v}_{k}){=}(\mathbf{u}^{(A_{k})},\mathbf{v}^{(A_{k})})$. Let $X$ be a discrete random variable denoting the index of the support that the AoA-AoD pair $\theta$ of the channel belongs to, so that $\theta{\in}\mathcal{B}^{({X})}$. Then, from (1)-(2), the received signal can be expressed as111The phase of $\mathbf{u}_{k}^{H}\mathbf{a}_{\mathrm{r}}(\theta_{\mathrm{r}})\mathbf{a}_{\mathrm{t}}^{H}(\theta_{\mathrm{t}})\mathbf{v}_{k}$ is incorporated into $\alpha_{k}$.

[TABLE]

where $\delta[\cdot]$ is the Kronecker’s delta function, equal to $1$ if alignment is achieved ($A_{k}{=}X$), equal to [math] otherwise ($A_{k}{\neq}X$); $G$ and $g$ are, respectively, the main and side lobe gains of the sectored model, expressed as

[TABLE]

[TABLE]

In the following, we describe the beam-alignment and data communication procedures.

Beam-Alignment: In each slot $k$ of the beam-alignment phase, the BS transmits a pilot sequence $\mathbf{s}$ using the beam index $A_{k}$, with transmit power $P_{\mathrm{tx},k}{=}P_{\mathrm{ba}}$. Upon receiving $\mathbf{z}_{k}$ (based on the combining vector with index $A_{k}$), the UE uses a matched filter to compute the signal strength and sends the normalized received power feedback signal $Y_{k}$ back to the BS, of the form

[TABLE]

where $\Lambda{\triangleq}\frac{P_{\mathrm{ba}}\|\mathbf{s}\|^{2}}{N_{0}W_{\mathrm{tot}}\ell(d)}$ is the pre-beamforming receive SNR during beam-alignment. Then, the probability density function (pdf) of $Y_{k}$ conditional on $(X,A_{k}){=}(x,a_{\mathrm{s}})$ is given by

[TABLE]

where $1/\nu$ is the mean signal power in case of alignment, with

[TABLE]

The BS uses a Bayesian approach to select $A_{k}$: starting from $\mathcal{H}_{0}{\triangleq}\emptyset$ and given the history of feedback and scanned beam indices $\mathcal{H}_{k}{\triangleq}\{(A_{j},Y_{j})\}_{j=0}^{k-1}$, the next beam index $A_{k}$ is selected. This procedure continues until the end of the beam-alignment phase.

Data communication: Upon completion of the beam-alignment phase, given the history of feedback and actions $\mathcal{H}_{L}$, the BS selects the data communication parameters: beam index for data communication $A_{\mathrm{d}}{{\in}\mathcal{I}}$, transmission power $P_{\mathrm{d}}{\in}[0,P_{\max}]$, and data rate $R_{\mathrm{d}}{\geq}0$. These parameters are used until the end of the data communication phase.

Let $b_{0}[x]$ be the prior belief over $X{=}x$ (or equivalently over $\theta{\in}\mathcal{B}^{(x)}$) available at the beginning of the beam-alignment phase. We define the expected rate during the communication phase (normalized by the frame duration), as

[TABLE]

where we have defined

[TABLE]

The probability term in (II) is the probability of achieving correct alignment, given the prior $b_{0}$ and the history $\mathcal{H}_{L}$ of feedback and actions during the beam-alignment phase, whereas the probability term in (8) denotes the probability of non-outage with respect to the realization of the fading process (i.i.d. over time), given that correct alignment has been achieved (we assume that mis-alignment yields outage with probability one, since $g\ll G$).

III Problem Formulation and Solution

We now formulate the beam-alignment and data communication problem in the context of a decision process. We define a policy $\mu$, part of our design, which operates as follows. At time $k$ during beam-alignment, given the history of feedback and actions $\mathcal{H}_{k}$, the BS selects the beam-alignment action $A_{k}{=}a_{\mathrm{s}}{\in}\mathcal{I}$ with probability $\mu_{k}(a_{\mathrm{s}}|\mathcal{H}_{k})$; given $\mathcal{H}_{L}$, the BS selects the data communication parameters as $(A_{\mathrm{d}},P_{\mathrm{d}},R_{\mathrm{d}})=\mu_{\mathrm{d}}(\mathcal{H}_{L})$. The goal is to design $\mu$ so as to maximize the expected communication rate, i.e.,

[TABLE]

where the expectation $\mathbb{E}_{\mu}$ is conditional on the prior belief $b_{0}$ and on the policy $\mu$ being executed during beam-alignment and data communication. Note that, using (II), we can rewrite the optimization problem as

[TABLE]

i.e., the problem can be decomposed into the following two independent problems:

1. find the optimal rate and power $(R_{d}^{*},P_{d}^{*})$ that maximize the expected rate in the communication phase, conditional on correct alignment being achieved ($X{=}A_{\mathrm{d}}$);
2. find the optimal beam-alignment policy and the beam index for communication $A_{\mathrm{d}}$ so as to maximize the probability of correct alignment. The first problem can be solved efficiently by maximizing (8). In the sequel, we consider the latter problem.

Let $b_{k}[x]{\triangleq}\mathbb{P}(X=x|\mathcal{H}_{k},b_{0})$ be the belief over $X{=}x$ given the history of actions and feedback and prior belief $b_{0}$. It serves as a sufficient statistic for optimal control for problem P1. In the following lemma, we present an equivalent simplified sufficient statistic along with its dynamics.

Lemma 1**.**

Let $m_{0}[x]{{\triangleq}}\ln b_{0}[x]$ denote the prior preference of $X=x$. Given the action and feedback pair $(A_{k},Y_{k})$, the belief at $k+1$ is updated as

[TABLE]

where

[TABLE]

and we have defined

[TABLE]

Proof.

Given the belief $b_{k}$ and $(A_{k},Y_{k})=(a_{\mathrm{s}},y)$, we have

[TABLE]

where (a) follows from the definition of belief; (b) follows from Bayes’ rule and $\propto$ denotes proportionality up to a normalization factor independent of $x$; (c) follows from the facts that $Y_{k}$ is independent of history $\mathcal{H}_{k}$ given $(X,A_{k})$, and $X$ is independent of action $A_{k}$ given $\mathcal{H}_{k}$, and by the definition of belief $b_{k}$; (d-e) follow by substitution of the pdf of $Y_{k}$ given in (5) and by definition of $J(y)$. We prove the lemma using induction. The lemma holds for $b_{0}$ by definition of $m_{0}$. Let $0\leq k\leq L-1$ and $b_{k}$ be given by (9), then using (III)(e) normalized to sum to one, we get

[TABLE]

where $m_{k+1}[x]$ is given by (10). ∎

Let $\mathbf{m}_{k}\triangleq[m_{k}[1],\ldots m_{k}[{|\mathcal{I}|}]$. Then, the previous lemma demonstrates that $\mathbf{m}_{k}$ is a sufficient statistic for control decisions, since it is sufficient for computing the belief $b_{k}$ at time $k$. Therefore, $\mu$ can be expressed as $A_{k}=\mu_{k}(\mathbf{m}_{k}),\ \forall 0\leq k\leq L$, which maps the current preference vector $\mathbf{m}_{k}$ to beam index $A_{k}\in\mathcal{I}$. This result makes it possible to achieve an efficient implementation, since the belief can be updated according to simple preference update rules as in (10), rather than via complex Bayesian belief updates. In the subsequent analysis, we will use $\mathbf{m}_{k}$ rather than $b_{k}$ as the state.

III-A MDP Formulation

Thanks to the identification of the sufficient statistic $\mathbf{m}_{k}$, we model the optimization problem P1 as a Markov decision process (MDP) and optimize the decision variables to maximize the alignment probability in the data-communication phase. The MPD is a 5-tuple $\langle\mathcal{T},\mathcal{S},\mathcal{I},f(\mathbf{m}_{k+1}|\mathbf{m}_{k},a_{k}),r_{k}(\mathbf{m}_{k},a_{k}),\forall k{\in}\mathcal{T}\rangle$, with elements described as follows.

Time Horizon: given as $\mathcal{T}{=}\{0,1,\ldots,L\}$ where $\mathcal{T}_{\mathrm{BA}}{\equiv}\mathcal{T}{\setminus}\{L\}$ denote the slot indices associated with the beam-alignment phase, whereas at $k{=}L$, the communication parameters are selected and used until the end of the frame.

State space: given as $\mathcal{S}=\mathbb{R}^{|\mathcal{I}|}$, i.e., all possible values of preference vectors $\mathbf{m}_{k}$.

Action space: the set containing all the beam indices, $\mathcal{I}$.

**State transition distribution: ** Given state $\mathbf{m}_{k}=\mathbf{m}$ and action $A_{k}=a_{\mathrm{s}}$ used in the $k$th stage of the beam-alignment phase, the feedback $Y_{k}=y$ is generated with pdf

[TABLE]

leading to the new state

[TABLE]

where $\boldsymbol{\delta}[a_{\mathrm{s}}]{=}[\delta[a_{\mathrm{s}},x]]_{\forall x\in\mathcal{I}}$ is the vector with entries $\delta[a_{\mathrm{s}},x]$.

Reward function: the reward is the probability of choosing a beam index such that $A_{\mathrm{d}}=X$ in the data communication phase, so that correct alignment is achieved, yielding

[TABLE]

We now formulate the value function iteration for the MDP.

III-B Value Function

The value function under the optimal policy is given as

[TABLE]

where $q_{k}$ is the Q-function under the state-action pair $(\mathbf{m},a)$, defined recursively as

[TABLE]

and for $k\in\mathcal{T}_{\mathrm{BA}}$, using (14),

[TABLE]

This yields the optimal value function in the data communication phase, by choosing the beam index with maximum preference ${A_{\mathrm{d}}}^{*}=\arg\max_{{A_{\mathrm{d}}}\in\mathcal{I}}m[{A_{\mathrm{d}}}]$,

[TABLE]

In the beam-alignment phase ($k\in\mathcal{T}_{\mathrm{BA}}$), combining (17) and (III-B), we obtain iteratively the value function as

[TABLE]

In the following theorem, whose proof is provided in the Appendix, we unveil structural properties of $V_{k}^{*}(\mathbf{m})$. We find a lower-bound and an upper-bound to the Q-function and show that these bounds are optimized by a policy which, in each stage of the beam-alignment phase, selects the beam index with the second-best preference. This result will be the basis for our proposed policy evaluated numerically in Sec. IV.

Theorem 1**.**

For $k\in\mathcal{T}_{\mathrm{BA}}$, the Q-function is bounded as

[TABLE]

where we have defined $\xi(a_{\mathrm{s}};\mathbf{m})$

[TABLE]

where

[TABLE]

Let $x_{[1]},x_{[2]},\dots,x_{[|\mathcal{I}|]}$ be an ordering of beam indices in decreasing order of preference, i.e., $m[x_{[1]}]\geq m[x_{[2]}]\geq\cdots,m[x_{[|\mathcal{I}|]}]$, then the optimal value function is bounded as

[TABLE]

with the maximizer of $q_{k}^{UB}$ and $q_{k}^{LB}$ given by the second-best beam index $x_{[2]}$.

Proof.

The proof is provided in the Appendix. ∎

As a result of this Theorem, both the upper and lower bounds of the Q-function are maximized by the second-best beam index policy, which selects the beam index with the second-best preference during the beam-alignment phase. This policy will be evaluated numerically in the next section, against other MAB-based schemes proposed in the literature.

IV Numerical Results

In this section, we evaluate the performance of the second-best beam index selection scheme ($a_{\mathrm{s}}{=}x_{[2]}$) with analog beamforming, and compare it with three other schemes. The first one is based on LTS, a popular MAB scheme [12]. In LTS, at each slot the action is chosen according to the belief distribution, i.e., $a_{\mathrm{s}}{\sim}b[x]$. The second scheme is based on scanning the most-likely beam index ($a_{\mathrm{s}}{=}x_{[1]}$) as proposed in [10] (first-best). The third scheme is based on UCB as proposed in [9]. We evaluate the performance of these three schemes in terms of the probability of alignment and spectral efficiency using Monte-Carlo simulation with $10^{5}$ iterations for each simulated point, with parameters as follows: $M_{\mathrm{t}}{=}128$, $M_{\mathrm{r}}{=}1$, $N_{0}{=}{-}174\mathrm{dBm/Hz}$, $W_{\mathrm{tot}}{=}200\mathrm{MHz}$, $T_{\mathrm{fr}}{=}20\mathrm{ms}$, $T_{s}{=}0.1\mathrm{ms}$, $f_{c}{=}30\mathrm{GHz}$, $d{=}10$m, $[\text{path loss exponent}]{=}2$. The BS uses $M_{\mathrm{t}}{=}128$ antennas and partitions the AoD space into $32$ sectors, each with a beamwidth of $\pi/32\mathrm{rad}$ and with uniform prior $b_{0}[x]=1/32,\ \forall x\in\mathcal{I}$; the UE is isotropic, hence it uses $M_{\mathrm{r}}{=}1$ antenna with a single sector. We use the beamforming design proposed in [13] for ULAs with antenna spacing $d_{\mathrm{t}}{=}\lambda/2$. With this configuration, the main-lobe and side-lobe gains are best approximated by $G\approx 14\mathrm{dB}$, $g\approx-11\mathrm{dB}$.

In Fig. 2, we depict the probability of alignment achieved by the aforementioned schemes versus the pre-beamforming SNR $\Lambda$. It can be observed that second-best has better performance than the other three schemes, with up to 7%, 10%, and 30% performance gains compared to first-best, LTS-based and UCB-based schemes. The performance gain of second-best is attributed to a better exploration-exploitation trade-off. The first-best scheme suffers from poor exploration since it ”greedily” chooses the beam index most likely to succeed, but fails to test other beams that may be under-explored, and is thus prone to make alignment errors. On the other hand, LTS-based scheme suffers from poor exploitation since it may scan least likely beams. The proposed second-best scheme, on the other hand, strikes a favorable trade-off between exploration and exploitation: instead of greedily choosing the most likely beam, it chooses the second most likely one, leading to better exploration than first-best; simultaneously, by not choosing beam pairs that are unlikely to succeed, it leads to a better exploitation compared to the LTS-based and UCB-based schemes. Finally, compared to UCB, second-best is better tailored to the structure of the model, since it aims to maximize the alignment probability at the end of the beam-alignment phase (see (16)), rather than the surrogate metric of UCB – the cumulative SNR accrued during beam-alignment.

In Fig. 3, we depict the spectral efficiency against the fraction of $T_{\mathrm{fr}}$ used for BA $LT_{\mathrm{s}}/T_{\mathrm{fr}}$. We fix the SNR for beam-alignment as $\Lambda=0\mathrm{dB}$ and the data-communication power as $P_{\mathrm{d}}^{*}{=}22\mathrm{dBm}$. Similar to Fig. 2, second-best outperforms the three other schemes, owing to improved alignment. The spectral efficiency is maximized at a unique maximizer $L^{*}$: it increases initially with $L{\leq}L^{*}$ as the beam-alignment probability improves with $L$. However, as $L$ increases beyond $L^{*}$, this gain is offset by the increased overhead and reduced duration of the data communication phase.

V Conclusions

In this paper, we have formulated the beam-alignment problem as a Bayesian MAB problem. For the optimal control design, we have identified a simplified sufficient statistic referred to as the preference of beam pairs. Based on the preference and bounding of the value function, we have proposed a heuristic policy, which selects the beam pair with the second best-preference to scan. We have shown numerically that the proposed scheme outperforms the first-best, LTS, and UCB based beam-alignment schemes proposed in the literature.

Appendix: Proof of Theorem 1

Proof.

We prove the theorem using induction. Notice that from the definition of Q-function (III-B) and the optimal value function expression (19) for $k=L$ , we get

[TABLE]

where we have defined the preference update (15) as

[TABLE]

Moreover, using (14) and (11) we note that

[TABLE]

This yields

[TABLE]

where (b) follows by evaluating the integral in (a) for the two cases in (22), and noting that it is given by $\xi(a_{\mathrm{s}};\mathbf{m})$. Using Lemma 2 and (Proof.)(b), the optimal value function becomes

[TABLE]

Thus, the theorem statement holds for $k{=}L-1$ with equality. Assume it holds for $k{+}1$. Using Lemma 2, we can bound

[TABLE]

[TABLE]

Using (III-B), the induction hypothesis (25) for $k{+}1$ and the above bound, we obtain $q_{k}(\mathbf{m},{a_{\mathrm{s}}})$

[TABLE]

Moreover, we note that

[TABLE]

By substituting (Proof.) and (27) into (Proof.), yields

[TABLE]

The first integral in (Proof.) is equal to $\xi(a_{\mathrm{s}};\mathbf{m})$ and the second integral is found to be equal to

[TABLE]

Upon substituting these integrals into (Proof.) yields the following lower-bound to the Q-function,

[TABLE]

which proves the induction step (20), and whose maximization (see Lemma 2) yields (25).

Similarly, using the induction hypothesis (26) for $k+1$ and the upper-bound

[TABLE]

we obtain the following upper-bound to the Q-function,

[TABLE]

The integral above is equal to $\xi(a_{\mathrm{s}};\mathbf{m})$, which proves the induction step (21), hence

[TABLE]

Noting that $\max_{a_{\mathrm{s}}}\xi(a_{\mathrm{s}};\mathbf{m})=\xi(x_{[2]};\mathbf{m})$ (see Lemma 2), and upon substitution in (32) yields (26). ∎

Lemma 2**.**

We have that $\arg\max_{a_{\mathrm{s}}\in\mathcal{I}}\xi(a_{\mathrm{s}};\mathbf{m})=x_{[2]}$ and

[TABLE]

Proof.

To show that $\arg\max_{a_{\mathrm{s}}\in\mathcal{I}}\xi(a_{\mathrm{s}};\mathbf{m})=x_{[2]}$, we proceed as follows. Clearly, if $a_{\mathrm{s}}\in\{x_{[2],}x_{[3]},\ldots,x_{[|\mathcal{I}|]}\}$, then $\max_{{\hat{a}}\neq a_{\mathrm{s}}}m[{\hat{a}}]{-}m[a_{\mathrm{s}}]=m[x_{[1]}]{-}m[a_{\mathrm{s}}]\geq 0>\ln(\nu)$, hence

[TABLE]

maximized at $a_{\mathrm{s}}{=}x_{[2]}$. Therefore, we restrict $a_{\mathrm{s}}\in\{x_{[1]},x_{[2]}\}$ without loss in performance. Next, we show that $\xi(x_{[2]};\mathbf{m})\geq\xi(x_{[1]};\mathbf{m})$. Let $\Delta{\triangleq}m[x_{[1]}]{-}m[x_{[2]}]$. If $\Delta{>}{-}\ln\nu$, then $\xi(x_{[1]};\mathbf{m})=e^{m[x_{[1]}]}$ and $\xi(x_{[2]};\mathbf{m})>\xi(x_{[1]};\mathbf{m})$. Otherwise,

[TABLE]

Note that $u(\Delta,\nu)$ is decreasing in $\Delta{\in}(0,-\ln\nu],{\forall}\nu{\in}(0,1)$, minimized at $\Delta{=}-\ln\nu$, yielding, after algebraic steps,

[TABLE]

In both cases, $\max_{a_{\mathrm{s}}}\xi(a_{\mathrm{s}})=\xi(x_{[2]})$. Upon substitution of $a_{\mathrm{s}}=x_{[2]}$ in (22), yields (33). ∎