Dynamic Content Updates in Heterogeneous Wireless Networks

TL;DR

This paper proposes a learning-based cache refreshment strategy for edge content storage in wireless networks to optimize user satisfaction while managing network costs.

Contribution

It introduces a novel cache refreshment approach that adapts to user content age tolerance using learning techniques, balancing QoS and network efficiency.

Findings

Enhanced user satisfaction through adaptive cache updates

Reduced network costs compared to frequent refresh strategies

Effective learning of user content age preferences

Abstract

Content storage at the network edge is a promising solution to mitigate the excessive traffic load due to on-demand streaming applications as well as to reduce the streaming delay. To this end, cache-enabled cellular architectures can be utilized to increase the provided quality-of-service (QoS) and to reduce the network cost. However, there are certain issues to be considered in the design of the content storage strategy such that the contents should be refreshed in order to responds user`s expectations. Using a frequent cache refreshment strategy the ratio of satisfied users can be increased at an increasing network cost. In this paper, we introduce a cache refreshment strategy via leveraging learning techniques so that users' tolerance to the age of content is learned and the content is refreshed accordingly.

Equations

$$h_n(t+1)=$$

$$\sum _{n=1}^N{\lambda }_n(t)=\lambda (t),$$

$${\lambda }_{nr}(t)+{\lambda }_{na}(t)={\lambda }_n(t).$$

$$\text{mathds}E[{\lambda }_n|\lambda ]=\lambda p_n,$$

$$\text{mathds}E[{\lambda }_{nr}|\lambda ,h_n]=\lambda p_n{P}_r^{(n)}(h_n).$$

$$C({\mathbf{\lambda }}_r(t),\mathbf{d}(t))=\sum _{n=1}^N{C}_n({\lambda }_{rn}(t),d_n(t)).$$

$$C_{BH}(\mathbf{d}(t))=\sum _{n=1}^N{C}_{BH}^{(n)}(d_n(t)),$$

$$C_n({\lambda }_{rn}(t),d_n(t))={\beta }_n+{\alpha }_n{\lambda }_{rn}(t)+d_n(t){\mathcal{E}}_n$$

$$\displaystyle\min_{\mathbf{d}(t)\in\{0,1\}^{N}}\lim_{T\rightarrow\infty}\frac{1}{T}\sum^{T}_{t=1}\mathds{E}\big{[}C(\bm{\lambda}_{r}(t),\mathbf{d}(t))\big{]}$$

$$V(\mathbf{h})+{\mu }^{\ast }=\underset{\mathbf{d}}{\min }{V}_{\mathbf{d}}(\mathbf{h}),$$

$$V_{\mathbf{d}}(\mathbf{h})=\bar{C}(\mathbf{h},\mathbf{d})+\text{mathds}P(\acute{\mathbf{h}}|\mathbf{h},\mathbf{d})V(\acute{\mathbf{h}}),$$

$$\bar{C}(\mathbf{h},\mathbf{d})=\sum _{\mathbf{\Lambda }}\text{mathds}P({\mathbf{\lambda }}_{\mathbf{r}}=\mathbf{\Lambda })C({\mathbf{\lambda }}_r,\mathbf{d})$$

$$V^{(n)}(h_n)+{\mu }_n^{\ast }=\underset{d_n}{\min }{V}_d^{(n)}(h_n)$$

$$V_d^{(n)}(h_n)={\bar{C}}_n(h_n,d_n)+d_n{V}^{(n)}(0)+(1-d_n)V^{(n)}(h_n+1)$$

$$V_{k+1}^{(n)}(h)$$

$$=\underset{d}{\min }(-{\mu }_n^{\ast }+{\bar{C}}_n(h_n,d_n)+d_n{V}_k^{(n)}(0)+(1-d_n)V_k^{(n)}(h+1))$$

$$\displaystyle\mu^{*}_{n}=\min_{H_{n}}\frac{1}{H_{n}+1}\bigg{(}\sum^{H_{n}}_{h=0}\bar{C}_{n}(h,0)+\mathcal{E}_{n}\bigg{)},$$

$${\bar{C}}_n(h,0)={\beta }_n+{\alpha }_n\lambda p_n{P}_{redirect}^{(n)}(h).$$

$$\underset{H_n}{\min }\left\{{\beta }_n+\frac{{\mathcal{E}}_n+{\alpha }_n{p}_n{\sum }_{h=0}^{H_n}{P}_{redirect}^{(n)}(h)}{H_n+1}\right\}.$$

$${\hat{C}}_n(H_n)=\frac{{\mathcal{E}}_n+{\sum }_{t=1}^{H_n}{C}_n({\lambda }_{rn}(t),0)}{H_n+1}$$

Full text

I Introduction

While proactive content caching has received significant interest in the recent years, most of the existing strategies in the literature (both with uncoded [1, 2, 3] and coded placement [4, 5]) have been designed under the assumption that content popularities are known in advance. Although it is possible to observe the global popularity of contents in on-demand video streaming services, such as YouTube [6], small-cell base stations (SBSs) usually serve a small geographical area, where the local content popularity might not be aligned with the global popularity [7]. This mismatch between the local and global content popularities requires the design of predictive caching policies that aim to learn the local content popularity from the user requests. Predictive caching policies can be classified into two main groups, namely predictive caching with unknown popularities [8, 9, 10] and predictive caching with time-varying popularities [11, 12, 13]. In [8, 9], the authors focus on a single SBS and model the predictive caching problem as a multi-arm bandit problem, in which the received user requests are utilized to predict the file popularities, and the optimal caching strategy is obtained by taking into account the cost of file replacements. In [10], this approach has been extended to a cooperative caching framework, where the SBSs fetch the requested content from the neighboring SBSs if the corresponding file is cached there. Another strategy for predictive caching follows a user-centric approach, possibly implemented at a higher layer, and records user requests for different contents as a matrix. Future user requests are predicted using matrix completion techniques by exploiting the correlations among the requests for different files, similarly to recommendation systems [14]. In [11], a cache replacement strategy has been introduced for time-varying popularity scenario to maximize the local service rate with a minimum replacement cost, while a more theoretical approach is taken in [13], which studies the cache update policy in the case of time-varying content popularities. In parallel to the aforementioned works, another relevant research direction is contextual predictive caching, where various features of the requests, such as genre of the video file, age of the user, or the time of the requests, are utilized to predict future requests and shape the caching strategy accordingly [15, 16]. Although the predictive caching framework is highly effective to increase the efficiency of edge caching, there are certain limitations. Most of the aforementioned predictive caching strategies are designed to predict only the content popularity. However, in many applications, e.g., news, weather, etc., freshness of the content is another important factor for the user satisfaction. Content caching and refreshment problem has been previously studied in [17, 18]. In this paper, we consider a heterogeneous cellular network with cache enabled SBSs and provide a cost aware content update policy. The proposed content update strategy consists of two parts; in the first part, we show that the structure of the optimal periodic content update policy that minimizes the network cost for given users’ tolerance to the age of the content is of threshold type, and in the second part, we utilize the multi-armed bandit approach to learn users’ tolerance to the age of the content. To the best of our knowledge, ours is the first work that utilizes the learning framework to analyze the user behavior based on the content age. Accordingly, we design an online cache refreshment policy to minimize the overall network cost.

II System model and problem formulation

We consider a cellular network with a macro base station (MBS) and a SBS serving the users in a cell. It is assumed that both the MBS and the SBS are equipped with cache memories, storing a library of $N$ distinct dynamic contents, denoted by $S_{1},\ldots,S_{N}$. We note that each dynamic content has a different popularity, i.e., the probability that a user requests content $S_{n}$ is $p_{n}$, for $n=1\ldots,N$.

II-A Content freshness

Dynamic contents, such as news videos, traffic and weather updates, may change frequently over time. While we assume that the MBS always has the fresh content updates, thanks to its relatively higher-bandwidth connection to the content server in the core network, the SBS needs to regularly refresh the dynamic contents in its cache to keep them up-to-date. For the SBS, downloading all the contents from the MBS through its limited backhaul link is costly in terms of energy, time and spectrum.

We consider a discrete time system model with equal-length time slots. At the beginning of each time slot the SBS decides on which contents to be updated. We assume that a content is updated at the beginning of the next time slot. The corresponding decision vector is denoted by $\mathbf{d}(t)=\left(d_{1},\ldots,d_{N}\right)$, i.e., when the application $S_{n}$ is updated at the end of time slot $t$, $d_{n}(t)=1$, and $d_{n}(t)=0$ otherwise.

We denote by $h_{n}(t)$ the age of the content $S_{n}$ in the SBS cache at time slot $t$. We assume a maximum age $T_{max}$ at which a content becomes obsolete. In other words, the age of a content increases until it becomes obsolete. Accordingly, age of content $S_{n}$, $n=1,\ldots,N$, evolves over time in the following way:

[TABLE]

We denote the length-$N$ vector of ages associated with all the dynamic contents in the library by $\mathbf{h}(t)$.

II-B User behavior

Let $\lambda(t)$ denote the number of users that request a content at time slot $t$. Whenever a user requests a content, the request is first off-loaded to the SBS to be served. Users have different tolerance levels to the age of the contents they receive. Hence, we consider that, with probability $P_{redirect}^{(n)}(h)$, a user is not satisfied with the age $h$ of content $S_{n}$, and thus, it places another request for content $S_{n}$. In that case, the new user request is served directly by the MBS with a fresh content. Let $\lambda_{n}(t)$ be the number of users that request content $n$, $n=1,\ldots,N$, in time slot $t$, which is governed by the popularity profile $p_{n}$. The number of users that request content $n$ is split into two disjoint sets, where the first set of users are redirected to the MBS, while the second set consists of the users satisfied with the service provided by the SBS. We denote these numbers by $\lambda_{rn}(t)$, $\lambda_{an}(t)$, respectively. Note that $\lambda_{rn}(t)$ and $\lambda_{an}(t)$ are governed by the random process $P_{redirect}^{(n)}(h_{n}(t))$.

Let $\bm{\lambda}_{r}(t)=(\lambda_{r1}(t),\ldots,\lambda_{rN}(t))$ be the vector associated with the number of redirected users for each content. We have the following:

[TABLE]

The corresponding expected values of these parameters are given as:

[TABLE]

II-C Decision model and the problem formulation

Let $C(\bm{\lambda}_{r}(t),\mathbf{d}(t))$ be the cost associated with serving the users redirected to the MBS at time $t$, and the backhaul cost associated with updating the contents, if there is any. In this work, we assume that this cost is linear in $\lambda_{rn}$111For example, in OFDMA, a user re-directed to the MBS is assigned a subcarrier, and the power allocated to that subcarrier adds linearly to the energy cost.. Hence,

[TABLE]

Similarly, we define the back-haul cost function $C_{BH}(\mathbf{d}(t))$ which is also a linear function.

[TABLE]

where $C^{(n)}_{BH}(d_{n}(t))=d_{n}(t)\mathcal{E}_{n}$, with $\mathcal{E}_{n}$ being the average back-haul cost of updating application $n$. If the SBS decides to update the content $n$ (or multiple contents), the age of the content is updated at the end of the time slot. Hence,

[TABLE]

Note that updating a content has an immediate cost which is larger than not-updating. However, the incurred extra cost in updating the content enables more users to be served at a local SBS.

We aim at minimizing the average total cost as follows:

[TABLE]

III MDP formulation

Define the state of the system to be $\mathbf{h}$. We denote by $V(\mathbf{h})$ the differential value function at state $\mathbf{h}$. The differential Bellman equations can be written as:

[TABLE]

where $\mu^{*}$ is the optimal average cost and $V_{\mathbf{d}}(\mathbf{h})$ is the differential action-value function defined by:

[TABLE]

where $\mathds{P}(\acute{\mathbf{h}}|\mathbf{h},\mathbf{d})$ is the transition probability from state $\mathbf{h}$ into $\acute{\mathbf{h}}$ when action $\mathbf{d}$ is taken which is governed by (1), and

[TABLE]

The MDP associated with the average cost minimization problem can be solved by well known value iteration algorithm. However, the cardinality of state space (i.e., $(T_{max})^{N}$) and action space (i.e., $2^{N}$) grow exponentially with the number of contents. Hence, the curse of dimensionality is the bottleneck for an efficient solution. To bypass this bottleneck, we note that the cost function in (6) is linear and the transition probabilities of each content does not affect the other. Hence, we can separate the value function in (10) into $N$ independent value functions each representing a distinct application. For each application $n$, we have

[TABLE]

We have developed a framework that has enabled distributed policies with respect to the individual contents. We will show that for each content, there exists a threshold policy on the age of the content for which it is optimal to update the content. The following lemma establishes the key property used to prove the structure of the optimal policy.

Lemma 1**.**

The differential value function $V^{(n)}(h_{n})$ for all $n=1,\ldots,N$ is non-decreasing with respect to the age of the content, $h_{n}$.

Proof.

We use the value iteration algorithm to prove the lemma. We start by an arbitrary $V^{(n)}_{0}(h)$ differential value function and obtain the $k$-step differential value function $V^{(n)}_{k}(h)$ as follows:

[TABLE]

Note that $\lim_{k\rightarrow\infty}V^{(n)}_{k}(h)=V^{(n)}(h)$. The proof is by induction. For $k=1$, $V^{(n)}_{1}(h)=\min_{d_{n}}(-\mu^{*}_{n}+\bar{C}_{n}(h_{n},d_{n})+d_{n}V^{(n)}_{0}(0)+(1-d_{n})V^{(n)}_{0}(h_{n}+1))$ which is the minimum of two non-decreasing functions, and thus, itself is a non-decreasing function in $h_{n}$. Assume that the lemma holds for $k$. Then according to (15), $V^{(n)}_{k+1}(h_{n})$ is also a non-decreasing function with respect to $h_{n}$. By letting $k\rightarrow\infty$, we conclude the proof by showing that $V^{(n)}(h_{n})$ is also a non-decreasing function in $h_{n}$. ∎

The lemma is intuitively clear considering the non-decreasing property of the cost functions and Bellman equations in (14).

Theorem 1**.**

For each content the optimal policy minimizing the average cost is a threshold policy.

Proof.

The monotonicity of the differential value functions prove the optimality of the threshold policy [19, Chapter 7]. Intuitively, due to the non-decreasing property of the $V^{(n)}(h_{n})$, at some age, it would be optimal to update the content. Since the differential value function is non-decreasing, a larger, or smaller age would not be able to yield a smaller average cost. ∎

IV Learning Content Popularity and Age Tolerance

In the previous section, we showed that the problem is separable and thus, the optimization can be performed for each content separately. Second, we proved that the policy minimizing the cost is a threshold policy. Hence, the SBS by monitoring the age of the contents individually, needs to optimize according to a single threshold for each content. Under the threshold policy, the age of a content increases linearly until it reaches the threshold wherein the content will be updated and the age will refresh to a value of zero. Thus the minimum cost associated with content $n$ is the solution of:

[TABLE]

where

[TABLE]

Considering the linearity of the cost functions, the average cost optimization becomes:

[TABLE]

The equivalent optimization problem depends on the redirection probabilities $P^{(n)}_{redirect}(h)$, that are unknown. Hence, in the following we resort to reinforcement learning methods to infer the redirection probabilities.

We consider a sequential learning framework in which the SBS at each iteration of the learning algorithm faces choosing a threshold $0\leq H_{n}\leq T_{max}$. After choosing the threshold, the SBS will observe a random cost associated with its decision;

[TABLE]

The learning algorithm should provide the SBS a method to adjust its strategy by observing the outcomes of its decision. This resembles the well-known multi-armed bandit (MAB) problem. In MAB, each action (i.e., thresholds) has an expected return value which is called value of that action. We denote the true value of action $H_{n}$ by q_{n}(H_{n})=\frac{1}{H_{n}+1}\bigg{(}\sum^{H}_{h_{n}=0}\bar{C}_{n}(h,0)+\mathcal{E}_{n}\bigg{)}. If the agent (i.e., SBS) knows $q(.)$ values, then it can simply choose the action with the minimum expected cost. Thus, we need an algorithm that can learn $q_{n}(.)$ values. A well-studied algorithm for learning those values is the $\epsilon$-greedy algorithm [20] which starts by an arbitrary estimate, $Q_{n}(.)$ about the value of the actions and interacts with the environment to update its initial estimates, eventually converging to the true estimates. Two critical phases associated with $\epsilon$-greedy algorithm is the exploitation and exploration stages. The agent utilizing the estimates greedily chooses an action, and thus, it exploits what it knows already. Meanwhile, if it chooses an action completely random regardless of the estimates we say that it explores. Exploitation is necessary to act upon the experience while exploration helps to improve the estimate values and it facilitates convergence to the true action values. The $\epsilon$-greedy algorithm is presented in Algorithm 1.

V Numerical Results

In this section, we aim at evaluating the performance of the $\epsilon$-greedy algorithm in finding the optimal thresholds that minimize the total cost of the system. Due to the separability, we consider only one content and we note that the learning processes for all the contents are the same. The popularity of the contents are modeled by a Zipf distribution with an exponent of $1.1$. We assume that on average a given user becomes dissatisfied with a content of age $h$ with probability of $e^{-0.4h}$. Users arrive at the system according to a Poisson distribution with rate $100$ users per time slot. The cost of re-directed users to MBS is $C_{n}(\lambda_{rn}(t))=10\cdot\lambda_{rn}(t)$ and the backhaul cost is assumed to be $500$. In Figure 2, we illustrate the performance of the $\epsilon$-greedy algorithm for $\epsilon=0,\ 0.05,\ 0.1$ by adopting average regret as the metric. The regret of a learning algorithm is defined to be the difference between the cost achieved by the learner and the optimal cost. Here, we obtain the optimal cost by assuming that $P_{redirect}(h)$ is known, and by numerically solving (16). Note that the estimates of the action values, $Q(H)$, is initialized to be [math] for all $H=0,\ldots,T_{max}$. Note also that, the greedy algorithm seems to achieve a better performance even if it always exploits. This is not too surprising considering that the action-values are initialized opportunistically (i.e., the initial costs are believed to be zero by the agent). At the beginning, i.e., $t=0$, the greedy algorithm believes that every action returns a value of zero. However, by trying each action it gets disappointed in that action and tries the rest. In other words the estimates are biased. Opportunistic initialization is a simple method to incentivize exploration. However, it can only happen once and at the beginning of time. This method quickly fails in non-stationary environments.

To show this, we also study the performance of the $\epsilon$-greedy algorithm in a time varying environment. We assume that at $t=300$ the backhaul cost decreases to a value of $400$. The results are depicted in Figure 3. We can see that the greedy algorithm cannot adapt to the non-stationary environment and it gets stuck in a sub-optimal threshold. Meanwhile, for $0.05$ and $0.1$-greedy algorithm, it is able to adapt to the environment thanks to their exploration strategy. A large value of $\epsilon$ results in more exploration, and thus, we can see that $0.1$-greedy algorithm has a faster decay in terms of the average regret compared to the $0.05$-greedy algorithm. However, note that $\epsilon$-greedy algorithm is expected to be at least $\epsilon$ away from the optimal cost. Thus, there is a trade-off between the rate of convergence and the value of convergence.

VI Conclusion

In this work, we developed a framework for a cost minimization problem in a dynamic content caching setting. Specifically, we aimed at striking a balance between the number of unsatisfied users whom are redirected to MBS and the cost of accessing the backhaul link by SBS in updating the dynamic contents. We formulated the cost minimization problem as an MDP and showed that the problem is separable with respect to the contests. Subsequently, we proved that a threshold policy in the age of the contents is optimal. In finding the optimal thresholds, we resorted to learning algorithms since users’ preferences are not known and vary with each content. To that extend, we represented the problem in MAB framework and through numerical results, we showed that it is possible to make the expected regret of the learning algorithm arbitrarily close to zero. The learning algorithm even shows adaptability to non-stationary settings. As a future work, we aim to investigate the system with non-linear cost functions. Under the non-linear cost functions, the problem is no longer separable and new solution methods needs to be investigated. As an extension of this work we will also analyze the heterogeneous cellular network architecture with energy harvesting SBSs.