Distributed Network Caching via Dynamic Programming

TL;DR

This paper develops optimal caching strategies for distributed networks with local storage, using dynamic programming to minimize costs considering content popularity, time, and location dynamics, aiming to improve network efficiency and user satisfaction.

Contribution

It introduces a novel optimization framework for distributed caching that accounts for spatio-temporal content dynamics and proposes a dynamic programming-based solution with reduced complexity.

Findings

The proposed algorithm effectively balances cache placement costs across space and time.

Simulation results demonstrate improved caching efficiency and cost reduction.

The framework adapts to changing content popularities and network conditions.

Abstract

Next-generation communication networks are envisioned to extensively utilize storage-enabled caching units to alleviate unfavorable surges of data traffic by pro-actively storing anticipated highly popular contents across geographically distributed storage devices during off-peak periods. This resource pre-allocation is envisioned not only to improve network efficiency, but also to increase user satisfaction. In this context, the present paper designs optimal caching schemes for \textit{distributed caching} scenarios. In particular, we look at networks where a central node (base station) communicates with a number of "regular" nodes (users or pico base stations) equipped with \textit{local storage} infrastructure. Given the spatio-temporal dynamics of content popularities, and the decentralized nature of our setup, the problem boils down to select what, when and \textit{where} to cache. To address this problem, we define fetching and caching prices that vary across contents, time and space, and formulate a global optimization problem which aggregates the costs across those three domains. The resultant optimization is solved using decomposition and dynamic programming techniques, and a reduced-complexity algorithm is finally proposed. Preliminary simulations illustrating the behavior of our algorithm are finally presented.

Equations

$$\text{(C1)}{s}_{t,m}^f=a_{t-1,m}^f,$$

$$\text{(C2)}{r}_{t,0}^f\le w_{t,0}^f+s_{t,0}^f+\sum _{m=1}^M\left({\underline{w}}_{t,m}^f.s_{t,m}^f\right).$$

$$\text{(C3)}{a}_{t,m}^f\le s_{t,m}^f+{\overline{w}}_{t,m}^f,\forall m=1\cdots M.$$

$$\text{(C4)}{a}_{t,0}^f\le s_{t,0}^f+w_{t,0}^f+\sum _{m=1}^M{\underline{w}}_{t,m}^f.s_{t,m}^f$$

$$\text{(C5)}{\overline{w}}_{t,m}^f\le s_{t,0}^f+w_{t,0}^f+\sum _{m^{\prime }\ne m}{\underline{w}}_{t,m^{\prime }}^f.s_{t,m^{\prime }}^f.$$

$$c_{t,m}^f:={\overline{\lambda }}_{t,m}^f{\overline{w}}_{t,m}^f+{\rho }_{t,m}^f{a}_{t,m}^f,\forall m=1\cdots M,$$

$$c_{t,0}^f:={\lambda }_{t,0}{w}_{t,0}^f+{\rho }_{t,0}^f{a}_{t,0}^f+\sum _{m=1}^M{\underline{\lambda }}_{t,m}^f{\underline{w}}_{t,m}^f,$$

$$\left({\mathbf{w}}_t^{\ast },{\mathbf{a}}_t^{\ast }\right):$$

$$\displaystyle=\!\!\!\!\!\underset{(\boldsymbol{w},\boldsymbol{a}){\textrm{ s.t. (C1)-(C5)}}}{\arg\mathop{\min}}\left\{c_{t}\left(\boldsymbol{a},\boldsymbol{w};\boldsymbol{\rho}_{t},\boldsymbol{\lambda}_{t}\right)+\underset{\boldsymbol{\theta}_{k}}{\mathbb{E}}\left[\underset{(\boldsymbol{w}_{k},\boldsymbol{a}_{k}){\textrm{ s.t. (C1)-(C5)}}}{\min}\sum\limits_{k=t+1}^{\infty}\gamma^{k-t}\left[c_{k}\left(\boldsymbol{a}_{k},\boldsymbol{w}_{k};\boldsymbol{\rho}_{k},\boldsymbol{\lambda}_{k}\right)\Big{|}{\boldsymbol{s}}_{t+1}={\boldsymbol{a}}\right]\right]\right\}$$

$$V\left(\mathbf{s},\mathbf{\theta }\right):$$

$$\bar{V}(\mathbf{s}):$$

$$\bar{\mathcal{C}}:=\mathbb{E}\left[\sum _{t=0}^{\infty }\sum _{f=1}^F{\gamma }^t{c}_t^f\left({\mathbf{a}}_t^f,{\mathbf{w}}_t^f;{\mathbf{\rho }}_t^f,{\mathbf{\lambda }}_t^f\right)\right]$$

$$\text{(P1)}\underset{\{({\mathbf{w}}_k^f,{\mathbf{a}}_k^f){\}}_{f,k\ge t}}{\min }$$

$$s.t.$$

$$\text{(P2)}\underset{(\mathbf{w},\mathbf{a})}{\min }$$

$$s.t.$$

$${\underline{w}}_{t,m}^{*}=\left\{\begin{array}[]{ll}1&\quad\text{if~{}}m=\arg\min_{m^{\prime\prime}\in{{\cal M}_{t}\cup\left\{{\textrm{Cloud}}\right\}}}{\underline{\lambda}}_{t,m^{\prime\prime}}\\ 0&\quad\text{for~{} all~{} other}~{}m\end{array}\right.$$

$$c_{\{a_{t,m}=1\}}=\left\{\begin{array}[]{ll}\rho_{t,m}+\gamma{\bar{V}}_{m}(1)&\quad\text{for~{}}m\in{\cal M}_{t}^{+}\\ \rho_{t,m}+{\overline{\lambda}}_{t,m}+\gamma{\bar{V}}_{m}(1)&\quad\text{for~{} all~{} other}~{}m\end{array}\right.$$

Full text

**Index Terms— ** Caching, Fetching, Dynamic programming, Value iteration, Dynamic pricing.

1 Introduction

The tremendous growth of data traffic over wired and wireless networks motivates the need for emerging technologies to meet the ever-increasing data demands [1]. Recognized as an appealing solution is the so-called caching, which refers to storing reusable popular contents across geographically distributed storage-enabled network entities [1, 2]. The rationale here is to smoothen out the unfavorable shocks of peak traffic periods by pro-actively storing anticipated highly popular contents at these storage devices and during off-peak instances. This idea of resource pre-allocation is envisioned to achieve significant network resource savings in terms of e.g., energy, bandwidth, and cost. Which already comes with user satisfaction [1, 2, 3].

Utilizing (stochastic) optimization approaches to enable content-agnostic storage-enabled network entities to pro-actively cache popular contents has quickly gained attention. Under static popularity settings, a multi-armed bandit formulation is considered in [4]. The coded, and convexified version is studied in [5]. Context and trend aware learning approaches are provided in [6, 7] and a transfer learning one in [8]. Belief propagation based distributed cooperative caching is investigated in [9], and an ADMM based distributed approach in [10]. Recently, by considering dynamic popularities, reinforcement learning approaches are utilized to cache contents [11, 12, 13].

Existing caching approaches accounting for dynamic popularities focus on enabling a single cache entity to decide (learn) what and when to cache. However, since caching is being carried out in a network level, to fully unleash its potentials a distributed approach is essential to efficiently utilize the globally available storage capacity over the entire network. Building on this observation, the present paper proposes optimal dynamic schemes to operate over a network where nodes, connected to a central controller, receive user requests and are equipped with local cache memories where they can store specific contents. In particular, the optimal cache-fetch decisions across caching entities will be found as the solution of a constrained optimization aimed at reducing an expected network cost aggregated across caching entities, contents and time instants. Since the caching decision at a given node not only affects the instantaneous cost but also the availability of the content in the next slots, the problem is indeed a dynamic programming (DP) one. DP algorithms typically involve functional estimation and, as a result, they are computationally expensive. A key aspect of this paper is to combine a marginalization and a decomposition approach to effectively reduce the dimensionality of the problem and, hence, limit the computational complexity.

2 Problem Formulation

Consider a distributed caching system composed of a central node (CN) connected to a number of regular nodes, all devised with local storage devices. The CN can be a base station in a cellular network, or a gateway router in a content delivery network. The nodes are then user devices or access points, correspondingly. We consider $m=1\cdots M$ nodes, $f=1,2\cdots F$ files, and $t=1,2,\cdots$ time instants. For convenience the CN, which is also connected to a cloud through a (congested) back-haul, is indexed by $m=0$. With this notation, at a given slot, every node receives user file requests as indicated by the binary variable $r_{t,m}^{f}$, where $r_{t,m}^{f}=1$ denotes that during slot $t$ file $f$ was requested at node $m$, and $r_{t,m}^{f}=0$ otherwise. Upon receiving a file request, the node forwards the request to the CN unless the file is already available in the cache of the node. If file $f$ is available at the CN, the content is served to node $m$. If not, the CN fetches required file either from the cloud or from one of the local nodes. Moreover, once the content has been delivered, all the nodes having the file must decide whether to keep it or to discard it. Decisions are made based on the state of the system as well as on the prices (costs) associated with caching and fetching, with the overall goal being to minimize the long-term aggregated cost.

2.1 Optimization variables and constraints

To formalize the setting, let us define the binary fetching variables ${\underline{w}}^{f}_{t,m}$, and $\overline{w}^{f}_{t,m}$, as well as the binary storing decision variable $a^{f}_{t,m}$. For the nodes, having ${\underline{w}}^{f}_{t,m}=1$, implies fetching file $f$ at slot $t$ from node $m$ to the CN, while ${\underline{w}}^{f}_{t,m}=0$ means “not fetching”. Similarly $\overline{w}^{f}_{t,m}$ indicates if the CN fetches the file $f$ from node $m$ at time $t$. To have a compact notation, fetching from the cloud to the CN is denoted as $w^{f}_{t,0}=\underline{w}^{f}_{t,0}=\overline{w}^{f}_{t,0}$. For all the nodes, the indicator $a^{f}_{t,m}=1$ means that file $f$ will be stored at node $m$ at the end of slot $t$, and the binary storage state variable $s^{f}_{t,m}$ accounts for availability of file $f$ at node $m$ at the beginning of slot $t$. The availability of file $f$ directly depends on the preceding caching decision, thus the first set of constraints is

[TABLE]

which must hold $\forall f,t,m$. Having $r^{f}_{t,m}=1$ and $s^{f}_{t,m}=0$ necessitates requesting the file from the CN, which in turn requires fetching it either from the cloud or from the nodes having the content. This gives rise to the second set of constraints

[TABLE]

The product ${\underline{w}}^{f}_{t,m}.s^{f}_{t,m}$ guarantees that fetching from a node is feasible only if the node has the file. Storing a file at a node necessitates to either the file being available locally or fetching it from the CN, giving rise to the following constraint

[TABLE]

Likewise, storing at the CN requires

[TABLE]

Finally, a node can fetch only if

[TABLE]

2.2 Prices and aggregated costs

As assumed earlier, both caching and fetching are costly. For node $m$, at time $t$ and for file $f$, let $\rho^{f}_{t,m}$, ${\underline{\lambda}}^{f}_{t,m}$, and ${\overline{\lambda}}^{f}_{t,m}$ stand for costs of storing, fetching to, and fetching from the CN, respectively. Then, the overall incurred cost is $c^{f}_{t}=\sum_{m=0}^{M}c^{f}_{t,m}$, where for regular nodes we have that

[TABLE]

while for $m=0$ (the CN) we have

[TABLE]

with $\lambda_{t,0}$ being the cost of fetching from the cloud and $\rho_{t,0}$ being the caching cost for the CN.

2.3 Problem Formulation

Let us collect all caching and fetching costs during slot $t$ for file $f$ at vectors $\boldsymbol{\rho}^{f}_{t}$ and ${\boldsymbol{\lambda}}^{f}_{t}=\left[\underline{\boldsymbol{\lambda}}^{f}_{t},\overline{\boldsymbol{\lambda}}^{f}_{t}\right]$, and define similarly the action vectors ${\boldsymbol{a}}^{f}_{t}$ and ${\boldsymbol{w}}^{f}_{t}=[\underline{\boldsymbol{w}}_{t},\overline{\boldsymbol{w}}_{t}]^{\top}$ as well as the request vector $\boldsymbol{r}_{t}^{f}$. The first step is to define the aggregated cost to be minimized. Since there is an inherent uncertainty in the state of the system, we consider the discounted long-term incurred average cost

[TABLE]

where $\gamma^{t}$ is an exponentially decaying weight which reduces the impact of distant (and therefore more uncertain) costs, and the expectation is taken with respect to the random variables $\{\boldsymbol{r}^{f}_{t},\boldsymbol{\rho}^{f}_{t},\boldsymbol{\lambda}^{f}_{t}\}_{f,t}$. This paper assumes these variables are stationary, so that the expectations can practically be estimated. In addition the knowledge of the state variables $\{\boldsymbol{r}^{f}_{t},\boldsymbol{\rho}^{f}_{t},\boldsymbol{\lambda}^{f}_{t}\}_{f,t}$ is causal, that is, their exact values are revealed at the beginning of every time slot $t$, and all the fetching and caching decisions are made sequentially. The goal is then to make sequential fetching-caching decisions that minimize the expected current plus future costs while satisfying the constraints (C1)-(C5). This gives rise to the following optimization problem

[TABLE]

Due to (C1), (P$1$) is indeed a DP, because current decisions will influence not only current, but also future costs. As a result, one needs to [14, p. 10]: (i) identify the current and the expected future aggregated cost (the latter gives rise to the so-called value function), (ii) write the Bellman optimality conditions; and (iii) suggest a method to estimate the value function. In addition, note that (P$1$) is indeed decomposable across files, since the fetching and caching prices are different for $f$, $t$, and $m$. In most practical setups, those prices are in fact (stochastic) Lagrange multipliers associated with constraints that have been handled using dual decomposition techniques (see, e.g., [15]). Definition of those prices for the investigated caching setup is certainly of interest, but out of the scope of this paper. For this reason, in the sequel the focus is on solving (P$1$) for a single file $f$. Hence, the superscript $f$ will be dropped. In addition, due to stationarity the subscript $t$ will be dropped occasionally as well.

3 DP-based optimal fetching and caching

To solve (P$1$), collect all the storage state variables in vector ${\boldsymbol{s}}_{t}$, define $\boldsymbol{\theta}_{t}=[{\boldsymbol{r}}_{t},{\boldsymbol{\rho}}_{t},{\boldsymbol{\lambda}_{t}}]$, and assume that their initial values ${\boldsymbol{s}}_{0}$ and ${\boldsymbol{\theta}}_{0}$ are given. Then the optimal fetch-cache decisions $({\boldsymbol{w}}^{\ast}_{t},{\boldsymbol{a}}^{\ast}_{t})$ are expressible as the solution to (8). The objective in (8) is then rewritten in (9) as the summation of current and discounted average future costs. The form of (9) is testament to the fact that problem (P$1$) is a DP and the caching decisions $\boldsymbol{a}$ influence not only the current cost $c_{t}(\cdot)$ but also future costs through the second term as well. Bellman equations can be leveraged for tackling such a DP. Under the stationarity assumption for variables $\{\boldsymbol{\theta}_{t}\}_{t}$, the term accounting for the future cost can be rewritten in terms of the * stationary value function* $V\left(\boldsymbol{s},\boldsymbol{\theta}\right)$ [14, p. 68]. This function, formally defined in (10), captures the minimum aggregated cost starting from “state” $(\boldsymbol{s},\boldsymbol{\theta})$. Assuming that prices and requests are i.i.d. across time slots and nodes, it can be shown that the solution for (P$1$) can also be found in terms of the reduced value function, ${\bar{V}}(\boldsymbol{s}):=\underset{{\boldsymbol{\theta}}}{\mathbb{E}}\left[V(\boldsymbol{s},\boldsymbol{\theta})\right]$, which depends only on $\boldsymbol{s}$ [16]. The reduced value function is important since not only enables solving (P$1$), but also alleviates the computationally burden of estimating $V\left(\boldsymbol{s},\boldsymbol{\theta}\right)$, which has a much larger dimensionality.

By rewriting the reduced value function recursively as the summation of the instantaneous cost and the aggregated future values, one readily arrives at the Bellman equations provided in (11). Thus, the problem at time $t$ reduces to

[TABLE]

which requires the estimation of the reduced value function $\bar{V}(\cdot)$. This can be done using the value iteration algorithm as tabulated in Alg. 1; for a detailed discussion cf. [14, p. 100]. Alternatively, if the distributions of the random parameters are unknown, stochastic online solvers based on $Q$-learning are well motivated [14, p. 148]. This is left as future work.

3.1 Discussing the solution to (P2)

To gain insights on the optimal fetching-caching solution, suppose that at time $t$, we have $s_{t,0}=0$ and the CN receives file request from node $m^{\prime}$. In such a case, $r_{t,0}=1$ and thus (C$2$) requires $w_{t,0}+\sum_{m\in{\cal M}_{t}}{\underline{w}}_{t,m}\geq 1$, where ${\cal M}_{t}$ denotes the set of nodes for which $s_{t,m}=1$. Since the reduced value function does not depend on the fetching decision ${\boldsymbol{w}}_{t}$ and the cost $c_{t}$ is linear in ${\underline{w}}_{t,m}$ [cf. (6) and (7)], we have that the optimal fetching is

[TABLE]

Deciding on the caching variable ${\boldsymbol{a}}_{t}$ is more complicated. Suppose first that the value function can be written as $\bar{V}(\boldsymbol{s})=\sum_{m=0}^{M}\bar{V}_{m}(s_{m})$ and that node $m^{\prime}$ was the one requesting the content. Then, we have that, the cost of setting $a_{t,m}=1$ is

[TABLE]

where ${\cal M}_{t}^{+}={\cal M}_{t}\cup\{0,m^{\prime}\}$. On the other hand, the cost of setting $a_{t,m}=0$ is just $c_{\{a_{t,m}=0\}}=\gamma{\bar{V}}_{m}(0)$ for all $m$. Hence, to obtain the optimal ${\boldsymbol{a}}_{t}$ one only needs to solve $(M+1)$ separate tests of the form $c_{\{a_{t,m}=1\}}\gtrless c_{\{a_{t,m}=0\}}$, one per node. However, in most cases the assumption of $\bar{V}(\boldsymbol{s})=\sum_{m=0}^{M}\bar{V}_{m}(s_{m})$ does not hold and, as a result, caching decisions are coupled across nodes. In addition, while for a small number of users (say 10 or less) both the estimation of ${\bar{V}}(\boldsymbol{s})$ and the computation of the multiple decision rule is affordable, the computational burden grows exponentially as $M$ increases. As a result, in networks with a large number of users, alternative schemes that try to impose additional structure on ${\bar{V}}(\boldsymbol{s})$ are well motivated.

4 Numerical tests

Here we run numerical simulations to assess the behavior of the developed caching policy. Due to space limitations, the focus will be on the behavior of the CN, but the analysis can be generalized to nodes with $m\geq 1$ as well. The caching cost $\rho_{0}$ is considered to be uniformly drawn from $[0,2\bar{\rho}_{0}]$. In addition, to model the fetching costs over the entire network, we define the “effective” fetching cost $\lambda_{\textrm{eff}}$ as the cost of fetching from any node or cloud to the CN. $\lambda_{\textrm{eff}}$ is also considered to be uniformly drawn from $[0,2\bar{\lambda}_{\textrm{eff}}]$. To assess the CN’s policy, let us define the caching ratio as the ratio of the number of caching decisions divided by the total number of decisions. Fig 1(a). depicts this ratio for different values of $\bar{\lambda}_{\textrm{eff}}$ and $\bar{\rho}_{0}$, for $s_{0}=1$, and $r_{0}=1$. The requests from the CN are assumed to follow a Bernoulli distribution with $p_{r}=0.5$. The plot reveals a non-symmetric cache-versus-fetch trade off curve for the CN’s policy, showing clearly how caching is preferred over fetching as the fetching cost increases. For small enough caching costs, the ratio goes to one (i.e., caching happens with probability $1$). Changing the initial state to $s_{0}=0$, and $r_{0}=0$ clearly affects the CN’s policy as depicted in Fig. 1(b). Now, even when caching is cheap, caching decisions are rare (always below $20$ %). Intuitively, this is due to the fact that the file has not already been requested ($r_{0}=0$) and, thus, caching is only made to take advantage of low fetching costs. Fig. 1(c) shows the caching ratio for $s_{0}=1$, $r_{0}=1$, and $p_{r}=0.05$. The results confirm the relevance of the popularity of a content, since for a wide range of caching costs fetching is always decided. Interestingly, even when fetching is costly, the algorithm tends to fetch. The reason is that, due to its low popularity, the file will have to remain for a long time in the memory before being used, entailing a very high aggregated caching cost.

We now compare the performance of the proposed method with that of simpler schemes. Specifically, Fig. 1(d) reports the sum average cost of the proposed DP approach compared with that of a myopic one. The myopic policy stores a file only if the file is requested or is locally stored and the current caching cost is less than the fetching one, i.e., $\rho_{0,t}<\lambda_{\textrm{eff},t}$. Here, $\bar{\rho}_{0}=10$, and as this result demonstrates the DP approach outperforms the myopic one as the difference between average costs increases. This is expected because in the myopic approach the file will be stored whenever the fetching is costly; however, the DP-based approach will also consider the probability of the file being requested in future instants. The next goal is to assess the benefits associated with distributed caching by comparing the obtained cost with the cost incurred when only the CN is equipped with a cache. To that end, Fig. 1(e) reports the results for different number of caching nodes $M\in\left\{0,1,2,4\right\}$. The fetching costs over any link of the network are assumed to be i.i.d, and uniformly drawn from $[0-2\bar{\lambda}]$. The caching costs are also considered i.i.d and drawn from $[0-2\bar{\rho}]$. This plot depicts the sum average cost incurred by the CN for $\bar{\rho}=62$, two values of $\bar{\lambda}$ ($40$ and $100$), and different probabilities of requesting the files. Clearly, increasing $M$ allows the CN to meet the demands and, hence, reduces the aggregated cost. As expected, this reduction saturates as $M$ increases. All in all, the results corroborate the benefits of having a system equipped with distributed caching infrastructure.

5 Conclusion

This paper illustrated how decomposition and dynamic programming tools can be used to design optimal fetching-caching schemes for a network of caches. In particular, we focused on an architecture formed by a central controller connected to the cloud and to a set of local nodes. At each time instant the controller received different content requests and had to decide: i) if the content was going to be downloaded from the cloud or from the local nodes (provided that it was available locally), and ii) if some of the local nodes should cache the content in their local memory for future use. Upon defining suitable fetching and caching costs (that varied across time, nodes and contents), the problem was formulated as a dynamic program and its optimal solution was obtained using different strategies to reduce the associated computational burden.