CBA: Contextual Quality Adaptation for Adaptive Bitrate Video Streaming (Extended Version)

TL;DR

This paper introduces CBA, a Bayesian contextual bandit algorithm for adaptive bitrate video streaming that optimally leverages client and network information to maximize QoE, especially in future network architectures like NDN.

Contribution

We propose a novel sparse Bayesian contextual bandit algorithm, CBA, for ABR streaming that effectively utilizes high-dimensional context data and is computationally efficient for online adaptation.

Findings

CBA outperforms existing algorithms in NDN emulation tests.

The fast inference scheme enables real-time adaptation.

CBA effectively balances client and network information for QoE maximization.

Abstract

Recent advances in quality adaptation algorithms leave adaptive bitrate (ABR) streaming architectures at a crossroads: When determining the sustainable video quality one may either rely on the information gathered at the client vantage point or on server and network assistance. The fundamental problem here is to determine how valuable either information is for the adaptation decision. This problem becomes particularly hard in future Internet settings such as Named Data Networking (NDN) where the notion of a network connection does not exist. In this paper, we provide a fresh view on ABR quality adaptation for QoE maximization, which we formalize as a decision problem under uncertainty, and for which we contribute a sparse Bayesian contextual bandit algorithm denoted CBA. This allows taking high-dimensional streaming context information, including client-measured variables and network assistance, to find online the most valuable information for the quality adaptation. Since sparse Bayesian estimation is computationally expensive, we develop a fast new inference scheme to support online video adaptation. We perform an extensive evaluation of our adaptation algorithm in the particularly challenging setting of NDN, where we use an emulation testbed to demonstrate the efficacy of CBA compared to state-of-the-art algorithms.

Figures

Equations

$$R_T=\sum _{t=1}^T{r}_t(a_t^{\ast })-\sum _{t=1}^T{r}_t(a_t),$$

$$q_t(a)=Q(1-\frac{1}{\alpha t},{\zeta }_{t-1}(a)),$$

$${\zeta }_{t-1}(a)=p({{\mathbf{x}}_t(a)}^{\top }\mathbf{\beta }(a)\mid {\mathcal{D}}_{t-1}(a)),$$

$${\mathcal{D}}_{t-1}(a)=\{({\mathbf{x}}_{t^{\prime }}(a),r_{t^{\prime }}(a),a_{t^{\prime }}):a_{t^{\prime }}=a,1\le t^{\prime }\le t-1\}.$$

$$q_t(a)={{\mathbf{x}}_t(a)}^{\top }{\mathbf{\mu }}_{\mathbf{\beta }}(a)+Q\left(1-\frac{1}{\alpha t},N(0,1)\right)\sqrt{{{\mathbf{x}}_t(a)}^{\top }{\mathbf{\Sigma }}_{\mathbf{\beta }(a)}{\mathbf{x}}_t(a)},$$

$$p(\mathbf{r}\mid \mathbf{\beta },{\sigma }^{-2})=\prod _{m=1}^{M}p(r_m\mid \mathbf{\beta },{\sigma }^{-2})=\prod _{m=1}^{M}N(r_m\mid {\mathbf{x}}_m^{\top }\mathbf{\beta },1/{\sigma }^{-2}),$$

$$\begin{array}{rl}& {\beta }_j\sim N({\beta }_j\mid 0,{\tau }_j/{\sigma }^{-2}),{\tau }_j\sim \mathrm{Gam}({\tau }_j\mid a_0,{\lambda }_j),\\ & {\lambda }_j\sim \mathrm{Gam}({\lambda }_j\mid b_0,\varphi ),\end{array}$$

$$\begin{array}{rl}\mathrm{KL}& (q(\mathbf{\beta },{\sigma }^{-2},\mathbf{\tau },\mathbf{\lambda },\varphi ,\omega )\Vert p(\mathbf{\beta },{\sigma }^{-2},\mathbf{\tau },\mathbf{\lambda },\varphi ,\omega \mid \mathbf{r}))\\ =& {\mathrm{E}}_q[\log (q(\mathbf{\beta },{\sigma }^{-2},\mathbf{\tau },\mathbf{\lambda },\varphi ,\omega ))]\\ & -{\mathrm{E}}_q[\log (p(\mathbf{\beta },{\sigma }^{-2},\mathbf{\tau },\mathbf{\lambda },\varphi ,\omega ,\mathbf{r}))]\\ & +\log (p(\mathbf{r})).\end{array}$$

$$\begin{array}{rl}\mathcal{L}(q)& ={\mathrm{E}}_q[\log (p(\mathbf{\beta },{\sigma }^{-2},\mathbf{\tau },\mathbf{\lambda },\varphi ,\omega ,\mathbf{r}))]\\ & -{\mathrm{E}}_q[\log (q(\mathbf{\beta },{\sigma }^{-2},\mathbf{\tau },\mathbf{\lambda },\varphi ,\omega ))]\\ & \le \log (p(\mathbf{r})).\end{array}$$

$$\begin{array}{rl}& q(\mathbf{\beta })=N(\mathbf{\beta }\mid {\mathbf{\mu }}_{\mathbf{\beta }},{\mathbf{\Sigma }}_{\mathbf{\beta }}),q({\sigma }^{-2})=\mathrm{Gam}({\sigma }^{-2}\mid c^{\ast },d^{\ast }),\\ & q(\mathbf{\tau })=\prod _{j=1}^D\mathrm{GIG}({\tau }_j\mid p_{\tau ,j},a_{\tau ,j},b_{\tau ,j}),\\ & q(\mathbf{\lambda })=\prod _{j=1}^D\mathrm{Gam}({\lambda }_j\mid a_{\lambda ,j},b_{\lambda ,j}),q(\varphi )=\mathrm{Gam}(\varphi \mid a_{\varphi },b_{\varphi }),\\ & q(\omega )=\mathrm{Gam}(\omega \mid a_{\omega },b_{\omega }),\end{array}$$

$$\begin{array}{rl}& {\mathbf{\mu }}_{\mathbf{\beta }}=({\mathbf{X}}^{\top }\mathbf{X}+{\mathbf{T}}^{-1}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{r}\\ & {\mathbf{\Sigma }}_{\mathbf{\beta }}=⟨{\sigma }^{-2}{⟩}^{-1}({\mathbf{X}}^{\top }\mathbf{X}+{\mathbf{T}}^{-1}{)}^{-1},\\ & {\mathbf{T}}^{-1}=\mathrm{diag}(⟨{\tau }_1^{-1}⟩,\dots ,⟨{\tau }_D^{-1}⟩),c^{\ast }=\frac{M+D+c_0}{2},\\ & d^{\ast }=({\mathbf{r}}^{\top }\mathbf{r}-2{\mathbf{r}}^{\top }\mathbf{X}⟨\mathbf{\beta }⟩+\sum _{m=1}^M{\mathbf{x}}_m^{\top }⟨\mathbf{\beta }{\mathbf{\beta }}^{\top }⟩{\mathbf{x}}_m\\ & +\sum _{j=1}^D⟨{\beta }_j^2⟩⟨{\tau }_j^{-1}⟩+d_0)/2,\\ & p_{\tau ,j}=a_0-1/2,a_{\tau ,j}=2⟨{\lambda }_j⟩,b_{\tau ,j}=⟨{\beta }_j^2⟩⟨{\sigma }^{-2}⟩,\\ & a_{\lambda ,j}=a_0+b_0,b_{\lambda ,j}=⟨{\tau }_j⟩+⟨\varphi ⟩,\\ & a_{\varphi }=D{b}_0+1/2,b_{\varphi }=⟨\omega ⟩+\sum _{j=1}^D⟨{\lambda }_j⟩,\\ & a_{\omega }=1,b_{\omega }=⟨\varphi ⟩+1\end{array}$$

$$\begin{array}{rl}& ⟨\mathbf{\beta }⟩={\mathbf{\mu }}_{\mathbf{\beta }},⟨\mathbf{\beta }{\mathbf{\beta }}^{\top }⟩={\mathbf{\Sigma }}_{\mathbf{\beta }}+{\mathbf{\mu }}_{\mathbf{\beta }}{\mathbf{\mu }}_{\mathbf{\beta }}^{\top },\\ & ⟨{\sigma }^{-2}⟩=c^{\ast }/d^{\ast },⟨{\lambda }_j⟩=a_{\lambda ,j}/b_{\lambda ,j},⟨\varphi ⟩=a_{\varphi }/b_{\varphi },\\ & ⟨\omega ⟩=a_{\omega }/b_{\omega },v_j=\sqrt{2a_{\tau ,j}{b}_{\tau ,j}},\\ & ⟨{\tau }_j⟩={\left(\frac{b_{\tau ,j}}{a_{\tau ,j}}\right)}^{1/2}\frac{{\mathrm{K}}_{p_{\tau ,j}+1}(v_j)}{{\mathrm{K}}_{p_{\tau ,j}}(v_j)},\\ & ⟨{\tau }_j^{-1}⟩={\left(\frac{a_{\tau ,j}}{b_{\tau ,j}}\right)}^{1/2}\frac{{\mathrm{K}}_{1-p_{\tau ,j}}(v_j)}{{\mathrm{K}}_{-p_{\tau ,j}}(v_j)},\end{array}$$

$$q({\theta }_m)=h({\theta }_m)\exp \{{\mathbf{\eta }}^{\top }\mathbf{S}({\theta }_m)-A(\mathbf{\eta })\}.$$

$${\hat{\nabla }}_{\mathbf{\eta }}\mathcal{L}=\tilde{\mathbf{\eta }}-\mathbf{\eta },$$

$${\mathbf{\eta }}^{(n+1)}={\mathbf{\eta }}^{(n)}+{\gamma }_n{\hat{\nabla }}_{\eta }\mathcal{L},$$

$${\mathbf{\eta }}^{(n+1)}=(1-{\gamma }_n){\mathbf{\eta }}^{(n)}+{\gamma }_n\hat{\mathbf{\eta }}.$$

$$\begin{array}{rl}& {\hat{\mathbf{\eta }}}_{\mathbf{\beta }}=\left(⟨{\sigma }^{-2}⟩M{\mathbf{x}}_m{r}_m,-\frac{1}{2}⟨{\sigma }^{-2}⟩(M{\mathbf{x}}_m{\mathbf{x}}_m^{\top }+{\mathbf{T}}^{-1})\right),\\ & {\hat{\mathbf{\eta }}}_{{\sigma }^{-2}}=(\frac{M+D+c_0}{2}-1,-(M{r}_m^2-2M{r}_m{\mathbf{x}}_m^{\top }⟨\mathbf{\beta }⟩\\ & +M{\mathbf{x}}_m^{\top }⟨\mathbf{\beta }{\mathbf{\beta }}^{\top }⟩{\mathbf{x}}_m+\sum _{j=1}^D⟨{\beta }_j^2⟩⟨{\tau }_j^{-1}⟩+d_0)/2),\\ & {\hat{\mathbf{\eta }}}_{\tau ,j}=\left(a_0-\frac{3}{2},-⟨{\lambda }_j⟩,⟨{\beta }_j^2⟩⟨{\sigma }^{-2}⟩/2\right),\\ & {\hat{\mathbf{\eta }}}_{\lambda ,j}=\left(a_0+b_0-1,-⟨{\tau }_j⟩-⟨\varphi ⟩\right),\\ & {\hat{\mathbf{\eta }}}_{\varphi }=\left(D{b}_0-\frac{1}{2},-⟨\omega ⟩-\sum _{j=1}^D⟨{\lambda }_j⟩\right),\\ & {\hat{\mathbf{\eta }}}_{\omega }=\left(0,-⟨\varphi ⟩-1\right).\end{array}$$

$$\begin{array}{rl}& \left({\mathbf{\mu }}_{\mathbf{\beta }},{\mathbf{\Sigma }}_{\mathbf{\beta }}\right)=\left(-\frac{1}{2}{{\mathbf{\eta }}_{\mathbf{\beta }}^{(2)}}^{-1}{\mathbf{\eta }}_{\mathbf{\beta }}^{(1)},-\frac{1}{2}{{\mathbf{\eta }}_{\mathbf{\beta }}^{(2)}}^{-1}\right),\\ & \left(c^{\ast },d^{\ast }\right)=\left({\eta }_{{\sigma }^{-2}}^{(1)}+1,-{\eta }_{{\sigma }^{-2}}^{(2)}\right),\\ & \left(p_{\tau ,j},a_{\tau ,j},b_{\tau ,j}\right)=\left({\eta }_{\tau ,j}^{(1)}+1,-2{\eta }_{\tau ,j}^{(2)},2{\eta }_{\tau ,j}^{(3)}\right),\\ & \left(a_{\lambda ,j},b_{\lambda ,j}\right)=\left({\eta }_{\lambda ,j}^{(1)}+1,-{\eta }_{\lambda ,j}^{(2)}\right),\\ & \left(a_{\varphi },b_{\varphi }\right)=\left({\eta }_{\varphi }^{(1)}+1,-{\eta }_{\varphi }^{(2)}\right),\\ & \left(a_{\omega },b_{\omega }\right)=\left({\eta }_{\omega }^{(1)}+1,-{\eta }_{\omega }^{(2)}\right)\end{array}$$

$$\begin{array}{rl}\text{QoE}& (s_t(a_t))=w_{1}v(a_t)-w_2[v(a_{t-1})-v(a_t){]}_{+}-w_{3}G(v(a_t))\end{array}$$

$$\begin{array}{rl}& \mathrm{GIG}(x\mid p,a,b)=(a/b{)}^{p/2}(2{\mathrm{K}}_p(\sqrt{ab}){)}^{-1}\\ & x^{p-1}\exp \{(ax+b/x)/2\}\end{array}$$

$$\begin{array}{rl}& p(\mathbf{\beta },{\sigma }^{-2},\mathbf{\tau },\mathbf{\lambda },\varphi ,\omega ,\mathbf{r})=p(\mathbf{r}\mid \mathbf{\beta },{\sigma }^{-2})p(\mathbf{\beta }\mid {\sigma }^{-2},\mathbf{\tau })\\ & p({\sigma }^{-2})p(\mathbf{\tau }\mid \mathbf{\lambda })p(\mathbf{\lambda }\mid \varphi )p(\varphi \mid \omega )p(\omega )\end{array}$$

$$q(\mathbf{\beta },{\sigma }^{-2},\mathbf{\tau },\mathbf{\lambda },\varphi ,\omega )=q(\mathbf{\beta })q({\sigma }^{-2})q(\mathbf{\tau })q(\mathbf{\lambda })q(\varphi )q(\omega )$$

$$\begin{array}{rl}& \mathcal{L}(q)=⟨\log p(\mathbf{r}\mid \mathbf{\beta },{\sigma }^{-2})⟩+⟨\log p(\mathbf{\beta }\mid {\sigma }^{-2},\mathbf{\tau })⟩\\ & +⟨\log p({\sigma }^{-2})⟩+⟨\log p(\mathbf{\tau }\mid \mathbf{\lambda })⟩+⟨\log p(\mathbf{\lambda }\mid \varphi )⟩\\ & +⟨\log p(\varphi \mid \omega )⟩+⟨\log p(\omega )⟩-⟨\log q(\mathbf{\beta })⟩\\ & -⟨\log q({\sigma }^{-2})⟩-⟨\log q(\mathbf{\tau })⟩-⟨\log q(\mathbf{\lambda })⟩\\ & -⟨\log q(\varphi )⟩-⟨\log q(\omega )⟩\end{array}$$

$$\begin{array}{rl}& ⟨\log (p(\mathbf{r}\mid \mathbf{\beta },{\sigma }^{-2}))⟩=-M/2\log (2\pi )+M/2⟨\log ({\sigma }^{-2})⟩\\ & -1/2⟨{\sigma }^{-2}⟩({\mathbf{r}}^{\top }\mathbf{r}-2{\mathbf{r}}^{\top }⟨\mathbf{\beta }⟩+\sum _{m=1}^M{\mathbf{x}}_m^{\top }⟨\mathbf{\beta }{\mathbf{\beta }}^{\top }⟩{\mathbf{x}}_m)\\ & ⟨\log (p(\mathbf{\beta }\mid {\sigma }^{-2},\mathbf{\tau }))⟩=-D/2\log (2\pi )+D/2⟨\log ({\sigma }^{-2})⟩\\ & -1/2\sum _{j=1}^D⟨\log ({\tau }_j)⟩-1/2\sum _{j=1}^D⟨{\sigma }^{-2}⟩⟨{\beta }_j^2⟩⟨{\tau }_j^{-1}⟩\\ & ⟨\log (p({\sigma }^{-2}))⟩=(c_0/2-1)⟨\log ({\sigma }^{-2})⟩-d_0/2⟨{\sigma }^{-2}⟩\\ & +c_0/2\log (d_0/2)-log(\Gamma (c_0/2))\\ & ⟨\log (p(\mathbf{\tau }\mid \mathbf{\lambda }))⟩=-D\log (\Gamma (a_0))+a_0\sum _{j=1}^D⟨\log ({\lambda }_j)⟩\\ & +(a_0-1)\sum _{j=1}^D⟨\log ({\tau }_j)⟩-\sum _{j=1}^D⟨{\lambda }_j⟩⟨{\tau }_j⟩\\ & ⟨\log (p(\mathbf{\lambda }\mid \varphi ))⟩=-D\log (\Gamma (b_0))+b_{0}D⟨\log (\varphi )⟩\\ & +(b_0-1)\sum _{j=1}^D⟨\log ({\lambda }_j)⟩-⟨\varphi ⟩\sum _{j=1}^D⟨{\lambda }_j⟩\\ & ⟨\log (p(\varphi \mid \omega ))⟩=\frac{1}{2}⟨\log (\omega )⟩-\log (\Gamma (\frac{1}{2}))-\frac{1}{2}⟨\log (\varphi )⟩\\ & -⟨\omega ⟩⟨\varphi ⟩\\ & ⟨\log (p(\omega ))⟩=\frac{1}{2}\log (\frac{1}{2})-\log (\Gamma (\frac{1}{2}))-\frac{1}{2}⟨\log (\omega )⟩\\ & -\frac{1}{2}⟨\omega ⟩\end{array}$$

$$\begin{array}{rl}& ⟨\log (q(\mathbf{\beta }))⟩=-\frac{D}{2}\log (2\pi )-\frac{1}{2}\log (|{\mathbf{\Sigma }}_{\mathbf{\beta }}|)\\ & ⟨\log (q({\sigma }^{-2}))⟩=c^{\ast }\log (d^{\ast })-\log (\Gamma (c^{\ast }))\\ & +(c^{\ast }-1)⟨\log ({\sigma }^{-2})⟩-d^{\ast }⟨{\sigma }^{-2}⟩\\ & ⟨\log (q(\mathbf{\tau }))⟩=(\frac{a_0}{2}-\frac{1}{2})[p\log (2)-p\log (⟨{\sigma }^{-2}⟩)\\ & +\sum _{j=1}^D\log (⟨{\lambda }_j⟩)-\sum _{j=1}^D\log (⟨{\beta }_j^2⟩)]-p\log (2)\\ & -\sum _{j=1}^D\log ({\mathrm{K}}_{a_0-\frac{1}{2}}(v_j))+(a_0-\frac{3}{2})\sum _{j=1}^D⟨\log ({\tau }_j)⟩\\ & -\sum _{j=1}^D⟨{\lambda }_j⟩⟨{\tau }_j⟩-\frac{1}{2}⟨{\sigma }^{-2}⟩\sum _{j=1}^D⟨{\beta }_j^2⟩⟨{\tau }_j^{-1}⟩\\ & ⟨\log (q(\mathbf{\lambda }))⟩=-D\log (\Gamma (a_0+b_0))+(a_0+b_0)\\ & \times \sum _{j=1}^D\log (⟨{\tau }_j⟩+⟨\varphi ⟩)+(a_0+b_0-1)\sum _{j=1}^D⟨\log ({\lambda }_j)⟩\\ & -\sum _{j=1}^D⟨{\tau }_j⟩⟨{\lambda }_j⟩-⟨\varphi ⟩\sum _{j=1}^D⟨{\tau }_j⟩\end{array}$$

$$\begin{array}{rl}& ⟨\log (q(\varphi ))⟩=(D{b}_0+\frac{1}{2})\log (⟨\omega ⟩+\sum _{j=1}^D⟨{\lambda }_j⟩)\\ & -\log (\Gamma (D{b}_0+\frac{1}{2}))+(D{b}_0-\frac{1}{2})⟨\log (\varphi )⟩\\ & -⟨\omega ⟩⟨\varphi ⟩-⟨\varphi ⟩\sum _{j=1}^D⟨{\lambda }_j⟩\\ & ⟨\log (q(\omega ))⟩=\log (⟨\varphi ⟩+1)-⟨\omega ⟩⟨\varphi ⟩-⟨\omega ⟩\end{array}$$

$$\begin{array}{rl}& \mathcal{L}=-\frac{M}{2}\log (2\pi )+\frac{c_0}{2}\log (\frac{d_0}{2})-\log (\Gamma (\frac{c_0}{2}))\\ & -c^{\ast }\log (d^{\ast })+log(\Gamma (c^{\ast }))+\sum _{j=1}^D⟨{\lambda }_j⟩⟨{\tau }_j⟩+⟨\varphi ⟩\sum _{j=1}^D⟨{\lambda }_j⟩\\ & -D\log (\Gamma (a_0))-D\log (\Gamma (b_0))-2\log (\Gamma (\frac{1}{2}))\\ & +\log (2)((\frac{5}{4}-\frac{a_0}{2})D-\frac{1}{2})+D\log (\Gamma (a_0+b_0))\\ & +\log (\Gamma (pD+\frac{1}{2}))+\frac{1}{2}⟨\omega ⟩+⟨\omega ⟩⟨\varphi ⟩+\frac{1}{2}\log (|{\mathbf{\Sigma }}_{\mathbf{\beta }}|)\end{array}$$

$$\begin{array}{rl}& +(\frac{a_0}{2}-\frac{1}{4})(D\log (⟨{\sigma }^{-2}⟩)-\sum _{j=1}^D\log (⟨{\lambda }_j⟩))\\ & +(\frac{a_0}{2}-\frac{1}{4})\sum _{j=1}^D\log (⟨{\beta }_j^2⟩)-\sum _{j=1}^D\log ({\mathrm{K}}_{a_0-\frac{1}{2}}(v_j))\\ & -\frac{1}{2}⟨{\sigma }^{-2}⟩\sum _{j=1}^D⟨{\beta }_j^2⟩⟨{\tau }_j^{-1}⟩-(D{b}_0+\frac{1}{2})\log (⟨\omega ⟩\\ & +\sum _{j=1}^D⟨{\lambda }_j⟩)-(a_0+b_0)\sum _{j=1}^D\log (⟨{\tau }_j⟩+⟨\varphi ⟩)-\log (⟨\varphi ⟩+1).\end{array}$$

$$\begin{array}{rl}& p(\mathbf{\beta }\mid \mathbf{r},\mathbf{X},{\sigma }^{-2},\mathbf{\tau })=N(\mathbf{\beta }\mid {\mathbf{\mu }}_{\mathbf{\beta }}^{\prime },{\mathbf{\Sigma }}_{\mathbf{\beta }}^{\prime })\\ & p({\sigma }^{-2}\mid \mathbf{r},\mathbf{X},\mathbf{\beta },\mathbf{\tau })=\mathrm{Gam}({\sigma }^{-2}\mid c^{\prime },d^{\prime })\\ & p(\mathbf{\tau }\mid \mathbf{\beta },{\sigma }^{-2},\mathbf{\lambda })=\prod _{j=1}^D\mathrm{GIG}({\tau }_j\mid p_{\tau ,j}^{\prime },a_{\tau ,j}^{\prime },b_{\tau ,j}^{\prime })\\ & p(\mathbf{\lambda }\mid \mathbf{\tau },\varphi )=\prod _{j=1}^D\mathrm{Gam}({\lambda }_j\mid a_{\lambda ,j}^{\prime },b_{\lambda ,j}^{\prime })\\ & p(\varphi \mid \mathbf{\lambda },\omega )=\mathrm{Gam}(\varphi \mid a_{\varphi }^{\prime },b_{\varphi }^{\prime })\\ & p(\omega \mid \varphi )=\mathrm{Gam}(\omega \mid a_{\omega }^{\prime },b_{\omega }^{\prime }),\end{array}$$

$$\begin{array}{rl}& {\mathbf{\mu }}_{\mathbf{\beta }}^{\prime }=({\mathbf{X}}^{\top }\mathbf{X}+{{\mathbf{T}}^{\prime }}^{-1}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{r},\\ & {\mathbf{\Sigma }}_{\mathbf{\beta }}^{\prime }=1/{\sigma }^{-2}({\mathbf{X}}^{\top }\mathbf{X}+{{\mathbf{T}}^{\prime }}^{-1}{)}^{-1},\\ & {{\mathbf{T}}^{\prime }}^{-1}=\mathrm{diag}({\tau }_1^{-1},\dots ,{\tau }_D^{-1}),\\ & c^{\prime }=\frac{M+D+c_0}{2},\\ & d^{\prime }=\frac{(\mathbf{r}-\mathbf{X}\mathbf{\beta }{)}^{\top }(\mathbf{r}-\mathbf{X}\mathbf{\beta })+{\mathbf{\beta }}^{\top }{{\mathbf{T}}^{\prime }}^{-1}\mathbf{\beta }+d_0}{2},\\ & p_{\tau ,j}^{\prime }=a_0-1/2,a_{\tau ,j}^{\prime }=2{\lambda }_j,b_{\tau ,j}^{\prime }={\beta }_j^2{\sigma }^{-2},\\ & a_{\lambda ,j}^{\prime }=a_0+b_0,b_{\lambda ,j}^{\prime }={\tau }_j+\varphi \\ & a_{\varphi }^{\prime }=D{b}_0+1/2,b_{\varphi }^{\prime }=\sum _{j=1}^D{\lambda }_j+\omega \\ & a_{\omega }^{\prime }=1,b_{\omega }^{\prime }=\varphi +1\end{array}$$

Full text

In this paper, we provide a fresh view on ABR quality adaptation for QoE maximization, which we formalize as a decision problem under uncertainty, and for which we contribute a sparse Bayesian contextual bandit algorithm denoted CBA. This allows taking high-dimensional streaming context information, including client-measured variables and network assistance, to find online the most valuable information for the quality adaptation. Since sparse Bayesian estimation is computationally expensive, we develop a fast new inference scheme to support online video adaptation. We perform an extensive evaluation of our adaptation algorithm in the particularly challenging setting of NDN, where we use an emulation testbed to demonstrate the efficacy of CBA compared to state-of-the-art algorithms.

I Introduction

Video streaming services such as Netflix, YouTube, and Twitch, which constitute an overwhelming share of current Internet traffic, use adaptive bitrate streaming algorithms that try to find the most suitable video quality representation given the client’s networking conditions. Current architectures use Dynamic Adaptive Streaming over HTTP (DASH) in conjunction with client-driven algorithms to adjust the quality bitrate of each video segment based on various signals, such as measured throughput, buffer filling, and derivatives thereof. In contrast, new architectures such as SAND [1] introduce network-assisted streaming via DASH-enabled network elements that provide the client with guidance, such as accurate throughput measurements and source recommendations. Given the various adaptation algorithms that exist in addition to client-side and network-assisted information, a fundamental question arises on the importance of this context information for the Quality of Experience (QoE) of the video stream.

The problem of video quality adaptation is aggravated in Future Internet architectures such as Named Data Networking (NDN). In NDN, content is requested by name rather than location, and each node within the network will either return the requested content or forward the request. Routers are equipped with caches to hold frequently-requested content, thereby reducing the round-trip-time (RTT) of the request while simultaneously saving other network links from redundant content requests. Several attempts to make DASH-style streaming possible over NDN exist, e.g., [2], for which the key difficulty is that traditional algorithms rarely play to the strengths of NDN where the notion of a connection does not exist. Throughput, for example, is not a trivial signal in NDN as data may not be coming from the same source.

In this paper, we closely look at the problem of using context information available to the client for video quality adaptation. Note that our problem description is agnostic to the underlying networking paradigm, making it a good fit to traditional IP-based video streaming as well as NDN. In essence, we consider the fundamental problem of sequential decision-making under uncertainty where the client uses network context information received with every fetched video segment. In Fig. 1 we show a sketch where the client adaptation algorithm decides on the quality of the next segment based on a high-dimensional network context. We model the client’s decision on a video segment quality as a contextual multi-armed bandit problem aiming to optimize an objective QoE metric that comprises (i) the average video quality bitrate, (ii) the quality degradation, and (iii) the video stalling.

One major challenge with incorporating high-dimensional network context information in video quality adaptation is extracting the information that is most relevant to the sought QoE metric. We note that the interactions within this context space become complicated given the NDN architecture, where the network topology and cache states influence the streaming session. Our approach introduces a sparse Bayesian contextual bandit algorithm that is fast enough to run online during video playback. The rationale behind the sparsity is that the given information, including network-assisted and client-side measured signals such as buffer filling and throughput, constitutes a high-dimensional context which is difficult to model in detail. Our intuition is that, depending on the client’s network context, only a few input variables have a significant impact on QoE. Note, however, that sparse Bayesian estimation is usually computationally expensive. Hence, we develop here a fast new inference scheme to support online quality adaptation.

Our contributions in this paper can be summarized as:

• •

We formulate the quality adaptation decision for QoE maximization in ABR video streaming as a contextual multi-armed bandit problem.

• •

We provide a sparse Bayesian contextual bandit algorithm, denoted CBA, which is computationally fast enough to provide real-world video players with quality adaptation decisions based on the network context.

• •

We show emulation testbed results and demonstrate the fundamental differences to the established state-of-the-art quality adaptation algorithms, especially given an NDN architecture.

The developed software is provided here111https://github.com/arizk/cba-pipeline-public. The remainder of this paper is organized as follows: In Sect. II, we review relevant related work on ABR video streaming and contextual bandits. In Sect. III, we present the relevant background on ABR video streaming. In Sect. IV, we model the quality adaptation problem as a contextual multi-armed bandit problem before providing a fast contextual bandit algorithm for high-dimensional information. In Sect. V, we show how ABR streaming uses CBA and define a QoE-based reward. We describe the evaluation testbed before providing emulation results in Sect. VI. Section VII concludes the paper.

II Related Work

In the following, we split the state-of-the-art related work into two categories; i.e., work on ABR quality adaptation, especially in NDN, and related work on contextual bandit algorithms with high-dimensional covariates.

Significant amounts of research have been given to finding streaming architectures capable of satisfying high bitrate and minimal rebuffering requirements at scale. CDN brokers such as Conviva [3] allow content producers to easily use multiple CDNs, and are becoming crucial to meet user demand [4]. Furthermore, the use of network assistance in CDNs has received significant attention recently as a method of directly providing network details to DASH players. SAND [1] is an ISO standard which permits DASH enabled in-network entities to communicate with clients and offer them QoS information. SDNDASH [5] is another such architecture aiming to maintain QoE stability across clients, as clients without network assistance information are prone to misjudge current network conditions, causing QoE to oscillate. Beyond HTTP, the capabilities of promising new network paradigms such as NDN pose challenges to video streaming. The authors of [2] compare three state-of-the-art DASH adaptation algorithms over NDN and TCP/IP, finding NDN performance to notably exceed that of TCP/IP given certain network conditions. New adaptation algorithms specific to NDN have also been proposed, such as NDNLive [6], which uses a simple RTT mechanism to stream live content with minimal rebuffering.

In this work, we model the video quality adaptation problem as a contextual bandit problem assuming a linear parametrization, which has successfully been used, e.g., for ad placement [7]. Another promising approach is based on cost-sensitive classification in the bandit setting [8]. Recently, [9] has discussed the use of variational inference in the bandit setting, wherein Thompson sampling is considered to cope with the exploration-exploitation trade-off. By assuming a high-dimensional linear parametrization, we make use of sparse estimation techniques. High-dimensional information arises in video streaming due to the network context. Sparsity has been a major topic in statistical modeling and many Bayesian approaches have been proposed. Traditionally, double exponential priors which correspond to $\ell_{1}$ regularization have been used. However, these priors often fail due to limited flexibility in their shrinkage behavior. Other approaches that induce sparsity include ’spike-and-slab’ priors [10] and continuous shrinkage priors. Between these two, continuous shrinkage priors have the benefit of often being computationally faster [11]. For our approach we use the Three Parameter Beta Normal (TPBN) continuous shrinkage prior introduced by [11], which generalizes diverse shrinkage priors, e.g, the horseshoe prior [12], the Strawderman-Berger prior, the normal-exponential-gamma prior, and the normal-gamma prior.

III Adaptive Bitrate Streaming: Decisions under Uncertainty

In this section, we review the established model for quality adaptation in ABR video streaming and highlight the changes that arise when streaming over NDN.

III-A Adaptive Bitrate Streaming: A Primer

In adaptive bitrate streaming, the content provider offers multiple qualities of the same content to clients, who decide which one to pick according to their own client-side logic. Each video is divided into $T$ consecutive segments which represents some fixed $L$ seconds of content. These segments are encoded at multiple bitrates corresponding to the perceived average segment quality. In practice, segment lengths are often chosen to be two to ten seconds [13] with several distinct quality levels to choose from, such as 720p and 1080p. Let $\mathcal{V}$ represent the set of all available video qualities, such that $\mathcal{V}=\{v(1),v(2),...,v(K)\}$ and $v(i)>v(j)$ for all $i>j$; i.e., a higher index indicates a higher bitrate and better quality. Let the $t$-th segment encoded at the $i$-th quality be denoted $s_{t}(i)$.

Received video segments are placed into a playback buffer which contains downloaded, unplayed video segments. Let the number of seconds in the buffer when segment $t$ is received be $B_{t}$, and let the playback buffer size BUF_MAX be the maximum allowed seconds of video in the buffer. By convention, we define $B_{0}=0$ and write the recursion of the buffer filling as $B_{t}=\max\{B_{t-1}+L-\xi_{t}(i),L\}$, where $\xi_{t}(i)$ denotes the fetch time for $s_{t}(i)$. A stalling event is ascribed to the $t$-th segment when $\xi_{t}(i)>B_{t-1}$. Note that the recursion above holds only if $B_{t-1}+L<\texttt{BUF\_MAX}$; i.e., the client is blocked from fetching new segments if the playback buffer is full. If this occurs, the client idles for exactly $L$ seconds before resuming segment fetching. In some related work [13], BUF_MAX is chosen between 10 and 30 seconds.

To allow the client to select a segment in the $i$-th quality, the client fetches a Media Presentation Description (MPD), an XML-like file with information on the available video segments and quality levels, during session initialization. After obtaining the MPD, the client may begin to request each segment according to its adaptation algorithm. In general, uncertainty exists over the segment fetch time. The most prevalent quality adaptation algorithms take throughput estimates [14] or the current buffer filling $B_{t}$ [15], or combinations and functions thereof to make a decision on the quality of the next segment $s_{t+1}(i)$. The decision aims to find the segment quality which maximizes a QoE metric, such as the average video bitrate, or compound metrics taking the bitrate, bitrate variations, and stalling events into account.

III-B Streaming over Named Data Networking

In NDN, consumers or clients issue interests which are forwarded to content producers, i.e., origin servers, via caching-enabled network routers. These interests are eventually answered with data provided by the producer or an intermediary router cache. To request a video, a consumer will first issue an interest for the MPD of the video. Each $s_{t}(i)$ is given a name in the MPD, e.g., of the form /video ID/quality level/segment number. The client issues an interest for each data packet when requesting a particular segment. Since NDN data packets are of a small, fixed size, higher-quality video segments will require more data packets to encode. We do not permit the client to drop frames, so all data packets belonging to some segment $s_{t}(i)$ must be in the playback buffer to watch that segment.

IV A Fast Contextual Bandit Algorithm for High Dimensional Covariates

In this work, we model the problem of video quality adaptation as a sequential decision-making problem under uncertainty, for which a successful framework is given by the multi-armed bandit problem dating back to [16]. The contextual bandit problem [17] is an extension to the classic problem, where additional information is revealed sequentially. The decision-making can therefore be seen as a sequential game.

At decision step $t$, i.e., at the $t$-th segment, a learner observes a $D$ dimensional context variable $\mathbf{x}_{t}(a)\in\mathbb{R}^{D}$ for a set of $K$ actions $a\in\{1,\dots,K\}$. Here, the actions map to the $K$ video qualities that the client chooses from. The client chooses an action $a_{t}$, for which it observes a reward $r_{t}(a_{t})$. This reward can be measured in terms of low-level metrics such as fetching time or, as we consider later, QoE. The decision making is performed over a typically unknown decision horizon $T$, i.e., $t\in\{1,\dots,T\}$. Therefore, the learner tries to maximize the cumulative reward $\sum_{t=1}^{T}r_{t}(a_{t})$ until the decision horizon. It is important to note that after each decision the learner only observes the reward $r_{t}(a_{t})$ associated to the played action $a_{t}$; hypothetical rewards for other actions $a\neq a_{t}$, are not revealed to the learner.

Next, we model the contextual bandit problem under the linearizability assumption, as introduced in [18]. Here, we assume that a parameter $\boldsymbol{\beta}^{\ast}(a)\in\mathbb{R}^{D}$ controls the mean reward of each action $a$ at decision step $t$ as $\operatorname*{\mathrm{E}}[r_{t}(a)]={\mathbf{x}_{t}(a)}^{\top}\boldsymbol{\beta}^{\ast}(a)$. We introduce the regret $R_{T}$ of an algorithm to evaluate its performance as

[TABLE]

with $a_{t}^{\ast}=\operatorname*{\arg\max}_{a}{\mathbf{x}_{t}(a)}^{\top}\boldsymbol{\beta}^{\ast}(a)$. The regret compares the cumulative reward of the algorithm against the cumulative reward with hindsight. In order to develop algorithms with a small regret in the linear setting, many different strategies have been proposed. Such algorithms include techniques based on forced sampling [19], Thompson sampling [20], and the upper confidence bound (UCB) [18, 7, 21, 22].

Network-assisted video streaming environments provide high-dimensional context information, so it is natural to assume a sparse parameter $\boldsymbol{\beta}^{\ast}(a)$. We therefore impose a sparsity-inducing prior on the sought regression coefficients $\boldsymbol{\beta}(a)$. To cope with the contextual bandit setting, we start with the Bayes-UCB algorithm with liner bandits introduced in [23] and develop a version which fits the given problem. Since previously developed sparse Bayesian inference algorithms are computationally expensive, we develop a fast new inference scheme for the contextual bandit setting.

IV-A The Contextual Bayes-UCB Algorithm - CBA

The Contextual Bayes-UCB algorithm (CBA-UCB) selects in each round the action $a$ which maximizes the index

[TABLE]

where $\alpha$ is a width parameter for the UCB and $Q(t,\rho)$ is the quantile function associated with the distribution $\rho$, i.e., $\mathbb{P}(X\leq Q(t,\rho))=t$, with $X\sim\rho$. Additionally, we denote $\zeta_{t-1}(a)$ as the posterior distribution of the mean reward

[TABLE]

where $\mathcal{D}_{t-1}(a)$ is the set of data points of contexts and rewards for which action $a$ was previously played

[TABLE]

In the following subsections, we derive a Gaussian distribution for the posterior distribution of the regression coefficients $p(\boldsymbol{\beta}(a)\mid\mathcal{D}_{t-1}(a))=\operatorname*{\mathcal{N}\,}(\boldsymbol{\beta}(a)\mid\boldsymbol{\mu}_{\boldsymbol{\beta}(a)},\boldsymbol{\Sigma}_{\boldsymbol{\beta}(a)})$. In this case the index in (2) reduces to

[TABLE]

where the quantile function computes to $\sqrt{2}\mathrm{erf}^{-1}(1-\frac{2}{\alpha t})$, with the inverse error function $\mathrm{erf}^{-1}(\cdot)$. The algorithm for CBA-UCB is depicted in Fig. 2, Alg. 1.

IV-B Generative model of the linear rewards

Here, we derive the posterior inference for the regression coefficients $\boldsymbol{\beta}(a)$. The posterior distributions are calculated for each of the $K$ actions. For the inference of the posterior (3), we use Bayesian regression to infer the posterior of the regression coefficients222For readability we drop the dependency on $a$ of the regression coefficients $\boldsymbol{\beta}(a)$ $\boldsymbol{\beta}=[\beta_{1},\dots,\beta_{D}]^{\top}$. We use the data $\mathcal{D}_{t-1}(a)$, which is a set of $M$ previously observed contexts $\mathbf{X}=[\mathbf{x}_{1},\dots,\mathbf{x}_{M}]^{\top}$ and rewards $\mathbf{r}=[r_{1},\dots,r_{M}]^{\top}$ when taking action $a$.

Assuming a linear regression model $\mathbf{r}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon},$ with i.i.d. noise $\boldsymbol{\epsilon}\sim\operatorname*{\mathcal{N}\,}(\boldsymbol{\epsilon}\mid\mathbf{0},\mathbf{I}_{M}/\sigma^{-2})$ the regression response $\mathbf{r}$ follows the likelihood

[TABLE]

where $\sigma^{-2}$ is the noise precision for the regression problem. For the application of video streaming with high-dimensional context information, we use a sparsity inducing prior over the regression coefficients $\boldsymbol{\beta}$ to find the most valuable context information. We use here the Three Parameter Beta Normal (TPBN) continuous shrinkage prior introduced by [11] , which puts on each regression coefficient $\beta_{j}$, $j\in\{1,\dots,D\}$, the following hierarchical prior

[TABLE]

where $\tau_{j}$ is a Gamma distributed333We use the shape and rate parametrization of the Gamma distribution. continuous shrinkage parameter that shrinks $\beta_{j}$, as $\tau_{j}$ gets small. The parameter $\lambda_{j}$ controls $\tau_{j}$ via a global shrinkage parameter parameter $\phi$. For appropriate hyper-parameter choice of $a_{0}$ and $b_{0}$ different shrinkage prior are obtained. For example we use $a_{0}=1/2$, $b_{0}=1/2$ which corresponds to the horseshoe prior [12]. For notational simplicity, we collect the parameters $\lambda_{j},\tau_{j}$ for the context dimensions $j\in\{1,\dots,D\}$ in the column vectors $\boldsymbol{\lambda},\boldsymbol{\tau}$, respectively.

For the estimation of the global shrinkage parameter $\phi$ an additional hierarchy is used as $\phi\sim\operatorname*{\mathrm{Gam}\,}(\phi\mid 1/2,\omega)$ and $\omega\sim\operatorname*{\mathrm{Gam}\,}(\omega\mid 1/2,1)$. For the noise precision a gamma prior is used $\sigma^{-2}\sim\operatorname*{\mathrm{Gam}\,}(\sigma^{-2}\mid c_{0}/2,d_{0}/2)$, with hyper parameters $c_{0}$ and $d_{0}$. The graphical model [24] of this generative model is depicted in Fig. 3.

IV-C Variational Bayesian Inference (VB)

In the following, we review the general approximate inference scheme of mean field variational Bayes (VB) and the application to the linear regression with TPBN prior as proposed in [11]. Thereafter, we leverage stochastic variational inference (SVI) to develop a new contextual bandit algorithm.

Since exact inference of the posterior distribution $p(\boldsymbol{\beta},\\ \sigma^{-2},\boldsymbol{\tau},\boldsymbol{\lambda},\phi,\omega\mid\mathbf{r})$ is intractable [25], we apply approximate inference in form of variational Bayes (VB) for posterior inference. We use a mean field variational approximation, with $q(\boldsymbol{\beta},\sigma^{-2},\boldsymbol{\tau},\boldsymbol{\lambda},\phi,\omega)=q(\boldsymbol{\beta})q(\sigma^{-2})q(\boldsymbol{\tau})q(\boldsymbol{\lambda})q(\phi)q(\omega)$ for the approximate distribution. The variational distributions are obtained by minimizing the Kullback-Leibler (KL) divergence between the variational distribution and the intractable posterior distribution

[TABLE]

By Jensen’s inequality, a lower bound on the marginal likelihood (evidence) can be found

[TABLE]

The evidence lower bound (ELBO) $\mathcal{L}(q)$ is used for solving the optimization problem over the KL divergence (7), since maximizing $\mathcal{L}(q)$ is equivalent to minimizing the KL divergence. Using calculus of variations [25], the solution of the optimization problem can be found with the following optimal variational distributions444$\operatorname*{\mathcal{G}\mathcal{I}\mathcal{G}\,}(x\mid p,a,b)$ denotes the generalized inverse Gaussian distribution, see Appendix A.

[TABLE]

with the parameters of the variational distributions

[TABLE]

and the moments

[TABLE]

where $K_{p}(\cdot)$ is the modified Bessel function of second kind.

The calculation of the ELBO $\mathcal{L}$ is provided in Appendix B. Fig 4 shows the probabilistic graphical model of the mean field approximation for the generative model. Note the factorization of the random variables which enables tractable posterior inference in comparison to the probabilistic graphical model for the coupled Bayesian regression in Fig. 3.

A local optimum of the ELBO $\mathcal{L}$ can be found by cycling through the coupled moments of the variational distributions. This corresponds to a coordinate ascent algorithm on $\mathcal{L}$. The corresponding algorithm is shown in Fig. 2 Alg. 3.

IV-D Stochastic Variational Inference (SVI)

Next, we present a new posterior inference scheme with TPBN prior based on stochastic variational inference (SVI) [26]. We optimize the ELBO $\mathcal{L}$ by the use of stochastic approximation [27] where we calculate the natural gradient that is obtained with respect to the natural parameters $\boldsymbol{\eta}$ of the exponential family distribution of the mean field variational distributions.

Consider the mean field approximation $q(\boldsymbol{\theta})=\prod_{m}q(\theta_{m})$ for the intractable posterior distribution $p(\boldsymbol{\theta}\mid\mathcal{D})$, where $\boldsymbol{\theta}$ and $\mathcal{D}$ denote the tuple of parameters and the data, respectively. For each factor $q(\theta_{m})$, assuming it belongs to the exponential family, the probability density is

[TABLE]

Here, $h(\theta_{m})$ denotes the base measure, $\boldsymbol{\eta}$ are the natural parameters, $\mathbf{S}(\theta_{m})$ is the sufficient statistics of the natural parameters, and $A(\boldsymbol{\eta})$ is the log-normalizer.

We compute the natural gradient of the ELBO $\mathcal{L}$ with respect to the natural parameters of the factorized variational distributions for each variational factor $q(\theta_{m})$. Therefore, the natural gradient computes to

[TABLE]

where $\tilde{\boldsymbol{\eta}}=\mathrm{E}_{q}[\boldsymbol{\eta}^{\prime}]$. The parameter $\boldsymbol{\eta}^{\prime}$ is the natural parameter of the full conditional distribution $p(\theta_{m}\mid\theta_{-m},\mathcal{D})$, where $\theta_{-m}$ denotes the tuple of all variables but $\theta_{m}$. Using a gradient update the variational approximation can be found as

[TABLE]

where $n$ denotes the iteration step of the algorithm and $\gamma_{n}$ is a step size parameter.

Random subsampling of the data enables constructing a stochastic approximation algorithm. For this, $\tilde{\boldsymbol{\eta}}$ is replaced by an unbiased estimate $\hat{\boldsymbol{\eta}}$, which yields a stochastic gradient ascent algorithm on the ELBO $\mathcal{L}$ in the form

[TABLE]

For the step size $\gamma_{n}$ we use $\frac{1}{n}$. In the case of the regression problem, we sample one data point $(\mathbf{x}_{m},r_{m})$ from the set of observed data points and replicate it $M$ times to calculate $\hat{\boldsymbol{\eta}}$. The intermediate estimates of the natural parameters are then obtained by

[TABLE]

The derivation is provided in Appendix C.

The transformation from the natural parametrization to the variational parametrization is calculated using

[TABLE]

and the moments can then be calculated with (11). We denote by $\eta^{(i)}$ the $i$-th variable of the tuple of natural parameters $\boldsymbol{\eta}$. The gradient update (14) with random subsampling is performed until the ELBO $\mathcal{L}$ converges. For an algorithmic description of SVI see Fig. 2 Alg. 2.

IV-E One Step Stochastic Variational Inference (OS-SVI)

Since the optimization of the ELBO $\mathcal{L}$ until convergence with both VB and SVI is computationally expensive, we present a novel one-step SVI (OS-SVI) algorithm for the bandit setting. In each round of OS-SVI the learner observes a context and a reward $(\mathbf{x}_{t}(a),r_{t}(a))$ based on the taken action $a_{t}=a$. This data point is used to update the variational parameters of the $a$-th regression coefficients $\boldsymbol{\beta}(a)$ by going one step in the direction of the natural gradient of the ELBO $\mathcal{L}$. For this we calculate the intermediate estimates (15) based on $t$ replicates of the observed data point $(\mathbf{x}_{t}(a),r_{t}(a))$. Thereafter, the stochastic gradient update is performed with (14). By transforming the natural parameters back to their corresponding parametric form (16), the updated mean $\boldsymbol{\mu}_{\boldsymbol{\beta}(a)}$ and covariance matrix $\boldsymbol{\Sigma}_{\boldsymbol{\beta}(a)}$ can be found. This update step is computationally significantly faster than using VB or SVI. The OS-SVI subroutine is described in Fig. 2 Alg. 4.

IV-F Accuracy and Computational Speed of the CBA-UCB Algorithms

For the numerical evaluation of the CBA-UCB with three parameter Beta Normal prior, we first create data based on the linearization assumption. We use a problem with decision horizon $T=1000$, $D=20$ dimensions, and $K=20$ actions. We use two experimental setups with a dense regression coefficient vector and a sparse regression coefficient vector, i.e., only five regression coefficients are unequal to zero.

We compare the developed algorithm CBA-UCB using the variants VB, SVI and OS-SVI with two base-line algorithms: LinUCB [7] and CGP-UCB [22]. For the CGP-UCB, we use independent linear kernels for every action. Fig. 5 and Fig. 6 show the average regret (1) for the dense and the sparse setting, respectively. For the sparse setting expected in high-dimensional problems such as network-assisted video streaming, CBA-UCB with VB yields the smallest regret. We observe in Fig. 5 that in the dense setting CGP-UCB obtains a high performance which is closely followed by CBA-UCB with VB. Note that CGP-UCB performs well, as Gaussian process regression with a linear kernel corresponds to a dense Bayesian regression with marginalized regression coefficients, and therefore matches the model under which the dense data has been created.

In Fig 1 we show the run-times of the algorithms, where we observe that the run-times for CBA-UCB with VB / SVI and the CGP-UCB baseline are impractically high. Further, this running performance deteriorates as the dimensions of the context grow, since the computational bottleneck of both VB and SVI are multiple matrix inversions of size $D\times D$, see Fig. 7. Fig. 8 shows the scaling of the run-time with the decision horizon $T$ with an identical setup as in Tab. 1. The CGP-UCB scales badly with $T$, as the kernel matrix of size $M_{a}\times M_{a}$ is inverted at every time step. Here, $M_{a}$ denotes the number of already observed contexts and rewards for decision $a$. Since the decision making has to be made in the order of a few hundred milliseconds for video streaming applications, neither CBA-UCB with VB nor CGP-UCB can be computed within this timing restriction. Therefore, we resort to the OS-SVI variant of the CBA algorithm, which empirically obtains a much smaller regret than the fast LinUCB baseline algorithm, but still retains a comparable run-time performance555For updating CBA-UCB with OS-SVI or LinUCB we only have to invert a $D\times D$ matrix once after a decision.. This renders the use of CBA with One Step Stochastic Variational Inference for network-assisted video quality adaptation feasible.

V Video Quality Adaptation as a Contextual Bandit Problem

In the following, we model ABR streaming as a contextual bandit problem where we use our developed CBA algorithm for video quality adaptation. The action set corresponds to the set of available bitrates $\mathcal{V}$ such that action $a_{t}\in\{1,\dots,K\}$ represents the decision to request quality $v(a_{t})$ for the $t$-th segment; i.e., to request the segment $s_{t}(a_{t})$. Below we formulate a real-valued segment-based QoE function to represent the reward $r_{t}(a_{t})$ obtained by performing $a_{t}$. Furthermore, we let $\mathbf{x}_{t}(a)$ represent the network context vector corresponding to an available action $a$ at segment $t$. At each $t$, therefore, there will be $K$ unique context vectors available.

V-A Online Video Quality Adaptation using CBA

CBA performs online video quality adaptation by calculating the index presented in (5) for each available action after observing the context vector $\mathbf{x}_{t}(a)$ of the action to determine the optimal bitrate to request for the next segment $t$. There are no constraints on the contents of the context vectors, allowing CBA to learn with any information available in the networking environment. Furthermore, each context feature may be either global or action-specific; for example, the current buffer filling percentage or the last 50 packet RTTs at bitrate $v(a)$, respectively. The action $a_{t}$ with the largest computed index is chosen, and a request goes out for $s_{t}(a_{t})$. Once $s_{t}(a_{t})$ is received, its QoE value below is calculated and fed to CBA as the reward $r_{t}(a_{t})$. CBA then updates its internal parameters before observing the next set of context vectors and repeating the process for segment $t+1$, until the video ends at segment $T$.

The performance of CBA depends upon several hyperparameters. In the description in Fig. 2, Alg. 1., we choose $\alpha=1$ as it was shown to yield the most promising results [23]. As mentioned in Sect IV, we use $a_{0}=b_{0}=1/2$ to obtain the horseshoe shrinkage prior. We let $c_{0}=d_{0}=10^{-6}$; we choose $c_{0}$ and $d_{0}$ to be small nonzero values such that a vague prior is obtained.

V-B Reward Formulation: Objective QoE

The calculated QoE metric is the feedback used by CBA to optimize the quality adaptation strategy. As QoE scores for a video segment may vary among users, we resort in this work to an objective QoE metric similar to [28] which is derived from the following set of factors:

Video quality: The bitrate of the segment. $v(a_{t})\in\mathcal{V}$. 2. 2.

Decline in quality: If the current segment is at a lower bitrate than the previous one, $[v(a_{t-1})-v(a_{t})]_{+}$ for two back to back segments666we use $[x]_{+}$ to denote $\max\{x,0\}$.. 3. 3.

Rebuffer time: The amount of time spent with an empty buffer after choosing $v(a_{t})$.

The rationale behind using the decline in quality, in contrast to the related work that counts quality variations, is that we do not want to penalize CBA if the player strives for higher qualities without risk of rebuffering. The importance of each component may vary based on the specific user or context, so, similar to [28], we define the QoE of a segment as a weighted sum of the above factors. Let the rebuffer time $G(v(a_{t}))$ be the amount of time spent rebuffering after choosing $v(a_{t})$. We define the QoE then as:

[TABLE]

where $w_{1}$, $w_{2}$, and $w_{3}$ are non-negative weights corresponding to the importance of the video quality, decline in quality, and rebuffer time, respectively. For a comparison of several instantiations of these weights, see [28].

Note that the above QoE metric is independent from CBA; the bandit is only given the scalar result of the calculation. CBA is able take arbitrary QoE metrics as specified input as long as these comprise a real-valued function to produce the reward metric.

VI Evaluation of Quality Adaptation in NDN

To evaluate the performance of CBA and compare it with Throughput-based (TBA) and Buffer-based (BBA) adaptation peers, we emulate the two NDN topologies: the doubles topology, shown in Fig. 9; and the full topology, shown in Fig. 10. The topologies are built using an extension of the Containernet project777https://github.com/containernet/containernet which allows the execution of Docker hosts as nodes in the Mininet emulator.

The NDN clients use a DASH player implemented with libdash, based on the code from [2] with Interest Control Protocol (ICP) parameters of $\gamma_{\text{ICP}}=2$, $\beta_{\text{ICP}}=0.5$, and $\texttt{initialWindow}=300$. We note that traffic burstiness can vary significantly depending on the ICP parameters used.

The clients begin playback simultaneously, where they stream the first 200 seconds of the BigBuckBunny video encoded in two-second H.264-AVC segments offered at the $K=5$ quality bitrates {1, 1.5, 2.1, 3, 3.5}Mbps, with a playback buffer size of 30 seconds. All containers run instances of the NDN Forwarding Daemon (NFD) with the access strategy, and repo-ng is used to host the video on the servers and caches.

In the following, we compare the performance of CBA in the VB and OS-SVI variants, in addition to the baseline algorithm LinUCB [7]. We also examine the performance of two state-of-the-art BBA and TBA algorithms, i.e., BOLA [15] and PANDA [14], respectively. There are many adaptation algorithms in the literature, some of which use BBA and TBA simultaneously, including [28], [29], [30], and [31]; however, BOLA and PANDA were chosen because they are widely used and achieve state-of-the-art performance in standard HTTP environments. Buffer filling percentage and quality-specific segment packet RTTs are provided to the client as context. Furthermore, we added a numHops tag to each Data packet to track the number of hops from the Data origin to the consumer.

We track the RTTs and number of hops of the last 50 packets of each segment received by the client in accordance with measurements from [32]. If a segment does not contain 50 packets, results from existing packets are resampled. As a result, each CBA algorithm is given a $D=101$ dimensional context vector constituted of the buffer fill percentage, packet RTTs, and numHops for each of the $K=5$ available qualities.

VI-A Results on the Doubles Topology

We modulate the capacity of the bottleneck link using truncated normal distributions. The link capacity is hence drawn with mean of 7Mbps, where it stays unchanged for a period length drawn with a mean of 5s. The weights in Eq. 17 are set to $w_{1}=6$, $w_{2}=2$, and $w_{3}=2$, emphasizing the importance of the average quality bitrate without allowing a large amount of rebuffering to take place. We note that the use of subjective quality evaluation tests for different users to map these weights to QoE metrics via, e.g., the mean opinion score (MOS), is out of the scope of this work.

Examining Tab. 1, we see that the one-step CBA-OS-SVI yields a significantly higher average bitrate. This is expected based on the QoE definition (17), but we might expect CBA-VB to pick high bitrates as well. However, we observe that the parameter update time for CBA-VB is 20 times greater than that of CBA-OS-SVI; this puts a delay of one-sixth of each segment length on average between receiving one segment and requesting another. Looking at CBA-VB in Fig. 11 we see that CBA-VB accumulates a much larger rebuffer time than other methods. Hence, CBA-VB is forced to request lower bitrates to cope with the extra rebuffer time incurred by updating its parameters. In addition, note that LinUCB fails to select high bitrates despite having a very small parameter update time, implying that LinUCB is not adequately fitting the context to the QoE and is instead accumulating a large amount of regret. This is corroborated by its cumulative QoE depicted in Fig. 11, which performs nearly as poorly as CBA-VB. By inducing sparsity on the priors and using just one sample, CBA-OS-SVI successfully extracts the most salient features quickly enough to obtain the highest cumulative QoE of all algorithms tested.

Interestingly, the CBA approaches shown in Fig. 1 also result in the lowest number of quality switches, though our QoE metric does not severely penalize quality variation. We see that the magnitude of their quality switches is also nearly half that of the other algorithms.

Concerning the rebuffering behavior, we observe rebuffering ratios of {4.5%, 8.4%, 11.4%, 17.6%, 32.9%} for LinUCB, BOLA, PANDA, CBA-OS-SVI, and CBA-VB, respectively. We trace some of the rebuffering events to the ICP congestion control in NDN. Note that tuning the impact of rebuffering on the adaptation decision is not a trivial task [2]. Fortunately, this is not hardwired in CBA but rather given through (17). Hence, in contrast to state-of-the-art adaptation algorithms, CBA could learn to filter the contextual information that is most important for rebuffering by tweaking the QoE metric used.

An important consideration when choosing a quality adaptation algorithm is fairness among clients while simultaneously streaming over common links. While this is taken care of in DASH by the underlying TCP congestion control, we empirically show here how the ON-OFF segment request behavior, when paired with the considered quality adaptation algorithms, impacts the QoE fairness in NDN. This is fundamentally different from considering bandwidth sharing fairness in NDN; e.g., in [2]. Here we are interested in QoE fairness since the QoE metric and not the bandwidth share is the main driver of the quality adaptation algorithm. Fig. 12 shows the regret of QoE fairness between both clients , where a larger regret indicates a greater difference in QoE between both clients up to a particular segment. Note that the regret is defined as a cumulative metric similar to (1). In accordance to the discussion in[33], the fairness measure used here is the entropy of the relative QoE of the two clients $H_{B}\left(\frac{\mathrm{QoE}_{\mathrm{client1}}(t)}{\mathrm{QoE}_{\mathrm{client1}}(t)+\mathrm{QoE}_{\mathrm{client2}}(t)}\right),$ where $H_{B}(\cdot)$ denotes the binary entropy and the QoE is given by (17). The regret is calculated with respect to the optimal fairness of $H_{B}^{\ast}\left(\frac{\mathrm{QoE}_{\mathrm{client1}}(t)}{\mathrm{QoE}_{\mathrm{client1}}(t)+\mathrm{QoE}_{\mathrm{client2}}(t)}\right)=1$. Observe that the CBA algorithms attain a significantly lower QoE fairness regret than other techniques.

VI-B Results on the Full Topology

To evaluate the capacity of CBA to adapt to different reward functions in complex environments, we compare performance with the full topology on two sets of weights in Eq. 17: HIGH_QUALITY_WEIGHTS sets $w_{1}=6$, $w_{2}=2$, and $w_{3}=2$, identical to those used in the evaluation on the doubles topology; conversely, NO_REBUFFERING_WEIGHTS sets $w_{1}=1$, $w_{2}=1$, and $w_{3}=3$, placing greater importance on continuous playback at the expense of video quality. We evaluate each algorithm with each weighting scheme for 30 epochs, where one epoch corresponds to streaming 200 seconds of the BigBuckBunny video. All clients use the same adaptation algorithm and weighting scheme within an epoch, and bandits begin each epoch with no previous context information.

Inspecting Tab. 2, we observe that the performance statistics among algorithms, even with different weighting schemes, are much closer than for the doubles topology. We attribute this to the use of a more complicated topology in which many more clients are sharing network resources, resulting in fewer and less predictable resources for each client. Furthermore, the average bitrate for the bandit algorithms does not change significantly across weighting schemes, and either stays the same or increases when using NO_REBUFFERING_WEIGHTS. This may seem contradictory, but, analyzing part (a) of Figs. 13 and  14, we note that CBA-OS-SVI tended to choose much lower bitrates with NO_REBUFFERING_WEIGHTS, and therefore accruing less rebuffer time in part (b), than with HIGH_QUALITY_WEIGHTS, indicating that CBA-OS-SVI successfully adapted to either weighting scheme within the playback window. Similarly to the doubles topology, LinUCB failed to map the context to either weighting scheme, selecting higher bitrates and rebuffering longer with NO_REBUFFERING_WEIGHTS. Note that, for either CBA-OS-SVI or LinUCB, the cumulative rebuffer time in part (b) of Figs. 13 and  14 tapers off roughly halfway through the video, as either algorithm learns to request more appropriate bitrates.

Interestingly, CBA-VB also fails to adapt to either weighting scheme, performing nearly identically in either case. This is a byproduct of the excessive parameter update time for CBA-VB in Tab. 2, which stems from the unpredictable nature of a larger network and the computational strain of performing up to 7 CBA-VB parameter updates simultaneously on the test machine. CBA-VB is therefore spending over half of the length of each segment deciding on which segment to request next, causing long rebuffering times in part (b) of Figs. 13 and  14, culminating in very low QoE scores regardless of the weighting scheme used. This obfuscates the underlying QoE function, preventing CBA-VB from differentiating between the weights in either case within the time allotted. In a real-world scenario, where each client is an independent machine, we expect that CBA-VB, as well as CBA-OS-SVI and LinUCB to a lesser extent, would have parameter update times comparable to those in the doubles topology, resulting in better performance; however, we note that evaluation in such an environment is out of the scope of this work.

Again, we see in Tab. 1 that CBA-OS-SVI switches qualities least frequently despite neither weighting scheme explicitly penalizing quality variation. Furthermore, according to parts (c) and (d) of Fig. 13 and Fig. 14, CBA-OS-SVI and CBA-VB are both stable in the number of quality switches and the quality switch magnitude across epochs, even under different weighting schemes, as opposed to the other algorithms tested.

VII Conclusions and Future Work

In this paper, we contributed a sparse Bayesian contextual bandit algorithm for quality adaptation in adaptive video streaming, denoted CBA. In contrast to state-of-the-art adaptation algorithms, we take high-dimensional video streaming context information and enforce sparsity to shrink the impact of unimportant features. In this setting, streaming context information includes client-measured variables, such as throughput and buffer filling, as well as, network assistance information. Since sparse Bayesian estimation is computationally expensive, we developed a fast new inference scheme to support online video quality adaptation. Furthermore, the provided algorithm is naturally applicable to different adaptive video streaming settings such as DASH over NDN. Finally, we provided NDN emulation results showing that CBA yields higher QoE and better QoE fairness between simultaneous streaming sessions compared to throughput- and buffer-based video quality adaptation algorithms.

Appendix A The Generalized Inverse Gaussian

The probability density function of a generalized inverse Gaussian (GIG) distribution is

[TABLE]

The GIG distribution with parameters $\boldsymbol{\theta}=[p,a,b]^{\top}$ is a member of the exponential family distribution with base measure $h(x)=1$, natural parameters $\boldsymbol{\eta}(\boldsymbol{\theta})=[p-1,-a/2,b/2]^{\top}$, sufficient statistics $\mathbf{S}(x)=[log(x),x,1/x]^{\top}$ and log-normalizer $A(\boldsymbol{\eta})=\log((-\eta^{(2)}/\eta^{(3)})^{(\eta^{(1)}+1)/2}\{2\mathrm{K}_{\eta^{(1)}+1}(\sqrt{-4\eta^{(2)}\eta^{(3)}})\}^{-1})$. The inverse transform of the natural parameters is obtained by $\boldsymbol{\theta}(\boldsymbol{\eta})=[\eta^{(1)}+1,-2\eta^{(2)},2\eta^{(3)}]^{\top}$.

Appendix B Calculation of the ELBO

Here, we present the calculation for the ELBO. The joint distributions involved in the calculation of the evidence lower bound (8) factorize as

[TABLE]

and

[TABLE]

Denoting $\langle\cdot\rangle$ as the expactation w.r.t. to the distribution $q$, the evidence lower bound (8) is

[TABLE]

The expected values of the log factorized joint distribution (19) needed for (21) are

[TABLE]

The expected values of the log factorized variational distribution (20) compute to

[TABLE]

[TABLE]

Therefore, we can calculate the evidence lower bound (8) as

[TABLE]

[TABLE]

Appendix C Calculation for the intermediate estimates of the natural parameters

In order to calculate (14) we need to calculate the intermediate parameters $\hat{\boldsymbol{\eta}}$ for all parameters, i.e., $\boldsymbol{\beta}$ , $\sigma^{-2}$, $\boldsymbol{\tau}$, $\boldsymbol{\lambda}$, $\phi$ and $\omega$. For this we calculate $\tilde{\boldsymbol{\eta}}=\mathrm{E}_{q}[\boldsymbol{\eta}^{\prime}]$, where $\boldsymbol{\eta}^{\prime}$ is the natural parameter of the full conditional for the corresponding variational factor.

Therefore, we compute the full conditionals for the parameters

[TABLE]

with the parameters

[TABLE]

Next, we transform the parameters of the full conditionals into the exponential family parametrization.

The inverse transform of the natural parameters of the full conditionals is given by

[TABLE]

Taking the expected value w.r.t. the variational distribution $q$ we calculate the parameters $\tilde{\boldsymbol{\eta}}=\mathrm{E}_{q}[\boldsymbol{\eta}^{\prime}]$ as

[TABLE]

Replicating one data point $(\mathbf{x}_{m},r_{m})$ $M$ times yields the intermediate estimates

[TABLE]