Locality

TL;DR

This paper proves that cache-oblivious algorithms that are asymptotically optimal in an ideal model are also optimal across various real-world systems that benefit from data locality, despite architectural differences.

Contribution

It establishes that cache-oblivious algorithms are asymptotically optimal in any system that rewards locality of reference, broadening their applicability.

Findings

Cache-oblivious algorithms are asymptotically optimal in real systems.

Optimality holds across systems with different cache architectures.

The result excludes some pathological cases.

Abstract

The program performance on modern hardware is characterized by \emph{locality of reference}, that is, it is faster to access data that is close in address space to data that has been accessed recently than data in a random location. This is due to many architectural features including caches, prefetching, virtual address translation and the physical properties of a hard disk drive; attempting to model all the components that constitute the performance of a modern machine is impossible, especially for general algorithm design purposes. What if one could prove an algorithm is asymptotically optimal on all systems that reward locality of reference, no matter how it manifests itself within reasonable limits? We show that this is possible, and that excluding some pathological cases, cache-oblivious algorithms that are asymptotically optimal in the ideal-cache model are asymptotically optimal in any reasonable setting that rewards locality of reference. This is surprising as the cache-oblivious framework envisions a particular architectural model involving blocked memory transfer into a multi-level hierarchy of caches of varying sizes, and was not designed to directly model locality-of-reference correlated performance.

Figures

Equations

$$Q_{\ell }^{\text{textsc}LoR}(n)=\{\begin{array}{ll}2\cdot Q_{\ell }^{\text{textsc}LoR}(\sqrt{n})+O(\log n)& \text{ if }n\ge 2\\ O(1)& \text{ otherwise }\end{array}$$

$$Q_{M,B}^{\mathbb{M}}(E(A,I))=\sum _{i=1}^{|E(A,I)|}\{\begin{array}{l}0\text{ if }{e}_i\in {\mathcal{W}}_{M,B}^{\mathbb{M}}(E(A,I),i-1)\\ 1\text{ otherwise}\end{array}$$

$${\exists }_{c>0,n_0>0}{\forall }_{n>n_0}{\forall }_{B\ge 1}{\forall }_{M\in {\mathcal{M}}_P(B)}{\forall }_{A^{\prime }\in {\mathcal{A}}_P}\left[W_{M,B}^{\text{textsc}co}(P,A,n)\le c\cdot W_{M,B}^{\text{textsc}co}(P,A^{\prime },n)\right]$$

$${\exists }_{c>0,n_0>0}{\forall }_{n>n_0}{\forall }_{B\ge 1}{\forall }_{M\in {\mathcal{M}}_P(B)}{\forall }_{A^{\prime }\in {\mathcal{A}}_P}\left[W_{M,B}^{\text{textsc}lru}(P,A,n)\le c\cdot W_{M,B}^{\text{textsc}lru}(P,A^{\prime },n)\right]$$

$$\forall_{I\in\mathcal{I}^{P}}\forall_{c>0}\forall_{B\geq 1}\forall_{M\in\mathcal{M}_{P}(B)}\Big{[}Q^{\textsc{co}}_{M,B}(E(A,I))=\Theta(Q^{\textsc{co}}_{c\cdot M,B}(E(A,I)))\Big{]},$$

$$\exists_{c>0,n_{0}>0}\forall_{n>n_{0}}\forall_{\ell\in\mathcal{L}}\forall_{A^{\prime}\in\mathcal{A}_{P}}{\Big{[}W^{\textsc{LoR}}_{\ell}(P,A,n)\leq c\cdot W^{\textsc{LoR}}_{\ell}(P,A^{\prime},n)\Big{]}}$$

$$\begin{gathered} {\exists }_{c>0,n_0>0}{\forall }_{n>n_0}{\exists }_{I_w\in {\mathcal{I}}_n^P}{\forall }_{B\ge 1}{\forall }_{M\in {\mathcal{M}}_P(B)} \\ \underset{A^{\prime }\in {\mathcal{A}}_P}{\min }{W}_{M,B}^{\text{textsc}co}(P,A^{\prime },n)\le c\cdot Q_{M,B}^{\text{textsc}co}(E(A,I_w)) \end{gathered}$$

$$\exists_{c>0,n_{0}>0}\forall_{n>n_{0}}\forall_{A^{\prime}\in\mathcal{A}_{P}}\exists_{I_{w}\in\mathcal{I}_{n}^{P}}\forall_{B\geq 1}\forall_{M\in\mathcal{M}_{P}(B)}\\ \Big{[}W^{\textsc{co}}_{M,B}(P,A,n)\leq c\cdot Q^{\textsc{co}}_{M,B}(E(A^{\prime},I_{w}))\Big{]}$$

$$Q_{M,B}^{\text{textsc}sco}(E)\le 2\cdot Q_{M,B}^{\text{textsc}co}(E)\le 4\cdot Q_{M,B}^{\text{textsc}sco}(E),$$

$$Q_{M,B}^{\text{textsc}slru}(E)\le 2\cdot Q_{M,B}^{\text{textsc}lru}(E)\le 4\cdot Q_{M,B}^{\text{textsc}slru}(E).$$

$$Q_{B,B}^{\text{textsc}co}(E(A,I))$$

$$=\sum _{i=1}^{|E(A,I)|}\{\begin{array}{ll}1& \text{if }\lfloor \frac{e_i}{B}\rfloor \ne \lfloor \frac{e_{i-1}}{B}\rfloor \\ 0& \text{otherwise.}\end{array}$$

$$Q_{\ell }^{\text{textsc}LoR}(E(A,I))=\sum _{i=1}^{|E(A,I)|}\ell (|e_i-e_{i-1}|)$$

$${\exists }_{c,n_0}{\forall }_{n>n_0}{\exists }_{I_w\in {\mathcal{I}}_n^P}{\forall }_{B\ge 1}$$

$${\exists }_{c^{\prime },n_0}{\forall }_{n>n_0}{\exists }_{I_w\in {\mathcal{I}}_n^P}{\forall }_{\ell \in \mathcal{L}}$$

$${\exists }_{c^{\prime },n_0}{\forall }_{n>n_0}{\forall }_{\ell \in \mathcal{L}}$$

$${\exists }_{c^{\prime },n_0}{\forall }_{n>n_0}{\forall }_{\ell \in \mathcal{L}}{\forall }_{A\in {\mathcal{A}}_P}$$

$$g(\delta )=g_x(\delta )=\{\begin{array}{l}0\text{ if }0\le \delta \le x\\ 1\text{ otherwise}\end{array}$$

$$\max \left(\left(\begin{array}{rl}& \stackrel{\text{left side}}{\overbrace{\underset{\begin{array}{c}\forall L<i\text{ s.t.}\\ e_L\le e_i\end{array}}{\min }\ell (e_i-e_L,{\delta }_t(E,i,L))}}\\ & +\\ & \underset{\text{right side}}{\underbrace{\underset{\begin{array}{c}\forall R<i\text{ s.t.}\\ e_R\ge e_i\end{array}}{\min }\ell (e_R-e_i,{\delta }_t(E,i,R))}}\end{array}\right)-1,0\right).$$

$$Q_{M,B}^{\text{textsc}slru}(E,i)$$

$$=\frac{1}{B}\sum _{s=0}^{B-1}1$$

$$=1$$

$$Q_{{}_{{\ell }_{M,B}}}^{\text{textsc}LoR}(E,i)$$

$$+{\ell }_{M,B}(e_R-e_i,{\delta }_t(E,i,R))-1,0)$$

$$=\max (1+1-1,0)=1$$

$$Q_{M,B}^{\text{textsc}slru}(E,i)$$

$$=\frac{e_R-e_i}{B}$$

$$Q_{{}_{{\ell }_{M,B}}}^{\text{textsc}LoR}(E,i)$$

$$+\min \left(1,\frac{e_R-e_i}{B}\right)-1,0)$$

$$=\max \left(1+\frac{(e_R-e_i)}{B}-1,0\right)$$

$$=\frac{(e_R-e_i)}{B}$$

$$Q_{M,B}^{\text{textsc}slru}(E,i)$$

$$=\frac{p(e_R)-B-p(e_L)}{B}$$

$$=\max \left(\frac{p(e_R)-p(e_L)}{B}-1,0\right)$$

$$=\max (\frac{B+e_R-e_i}{B}$$

$$-\frac{B-(e_i-e_L)}{B}-1,0)$$

Full text

1 Introduction

Modeling memory access time of modern computers is an important area of research that lies at the intersection of theoretical computer science, algorithm engineering, and practical aspects of computing. The reality is that modern computers are extremely complicated with numerous components that try to reduce the access time of the elements in the memory. Consequently, the access time varies by orders of magnitude depending on whether favorable conditions are met or not. Choices in the algorithm design can highly impact reaching those favorable conditions, which necessitates building good theoretical models of memory structures of modern computers.

The main direction of existing theoretical work has been on modeling of the memory hierarchy. This has been initiated by the Disk Access Model (DAM) [3], which is also known as the I/O model or the External Memory (EM) model. The DAM assumes a (fast) memory of size $M$ and a disk (i.e., slow memory) of infinite size. The disk stores the input in blocks of size $B$ and can be read or written to via the input-output (I/O) operations, where each I/O transfers one block of data at unit cost. The analysis then typically only considers the number of I/Os (read or written), ignoring any other computational cost. The justification is that since a disk is so much slower than internal memory, minimizing the number of block transfers and ignoring all else is a good model of runtime, however, sometimes this is not a realistic assumption. For example, when DAM is used to model cached memory (modeling cache misses as I/O operations, $M$ as the size of the cache, and $B$ as the size of the cache lines), the relative difference in the cost of a cache misses compared to the arithmetic operations is much smaller. Aside from this, the DAM model has additional limitations, e.g., it ignores the fact that accessing adjacent blocks on a disk is in practice much faster than two random blocks [13] and it models only two levels of memory.

Modeling more than two levels of the memory hierarchy is rather challenging. The big problem is that very precise models (e.g., the ones defining individual parameters for each level of memory hierarchy [14]) are often too complicated, making it hopelessly difficult to design and analyze algorithms. Other approaches, such as the hierarchical memory model (HMM) [1, 2], model memory with variable access costs by assuming that the cost to access a memory address $x$ is a non-decreasing function, $f(x)$, of the address itself. However, this does not accurately represent modern caches.

The most successful attempt at analyzing cache misses in multi-level cache hierarchies is probably the cache-oblivious framework [9]. It surprisingly avoids the complexity of modeling memory hierarchies by completely avoiding it: the algorithms are designed “obliviously to $M$ and $B$”, i.e., in the classical (single-level) RAM model. However, if such algorithms are analyzed in the two-level DAM with the best cache-management policy, also known as the ideal-cache model, and they happen to be efficient in terms of cache misses with respect to the two levels of memory, then they are also efficient for all levels of multi-level memory hierarchy. Moreover, it has been shown [12] that such algorithms are also efficient for many reasonable cache management policies, e.g., least-recently used (LRU) policy, typically implemented by hardware in practice. However, up to now, cache-oblivious algorithms haven’t been shown to optimize anything beyond cache misses.

Capturing locality of reference. In a real hardware, cache utilization is only one aspect that affects program runtime. For example, Jurkiewicz and Mehlhorn [10] show that the time it takes to perform address translations for virtual memory noticeably affects the runtime of programs on real hardware. Figure 1 illustrates the complicated ways various hardware features affect the memory access time. It shows a memory hierarchy, where cheaper and larger but slower memories are placed at the top. The functions that describe the access times of the components may have different behaviors, e.g., for a magnetic tape the access time is basically a linear function of the physical distance, whereas for hard disks it is a more complicated function.

Locality of reference is a fundamental principle of computing that heavily impacts both hardware and algorithm design [13]. In his widely-cited article [8], Peter Denning provides an overview of the history of the concept and how it became very popular in almost all aspects of computing. To quote him, “The locality was adopted as an idea almost immediately by operating systems, database, and hardware architects.”

The DAM algorithm and the cache-oblivious algorithms try to capture spatial locality and temporal locality – the two fundamental components of the notion of “locality of reference” – from an algorithmic point of view. However, given the complexity of modern hardware, the main question is if other, possibly more complex hardware features, can be modeled simply enough to facilitate design and analysis of algorithms. The approach of the HMM model [1, 2] to model the cost of access as a function of memory address is one way to keep the modeling complexity at bay. For instance the authors [1, 2] show that if the cost of accessing address $x$ is $\log(x)$ then sorting $n$ elements can be done in $O(n\log n\log\log n)$ time. But is $\log(x)$ the correct function for modern (and ever-changing) hardware?

1.1 Our Results.

We propose to pick up where the previous attempts have left off by following a holistic approach. We present the locality of reference (LoR) model, a computational model that looks at memory in a new way: the cost of a memory access is based on the proximity from prior accesses via what we call a locality function.

More specifically, we consider the machine as having an infinite memory with a linear address space, i.e., memory cells are numbered with the set of natural numbers. Let $E=\{e_{1},\ldots,e_{|E|}\}$ be the sequence of memory addresses accessed by an algorithm $A$ while running on a given input. A simple locality function $\ell$ can define the cost of accessing address $e_{i}$ as a function of the distance from the address $e_{i-1}$ of the preceding access, for example, $\log(|e_{i}-e_{i-1}|)$, $\sqrt{|e_{i}-e_{i-1}|}$, or any other arbitrarily function of $|e_{i}-e_{i-1}|$.

A specific locality function can capture the (complicated) cost of accessing the data on the hardware running the algorithm. For example, in the classical random access memory (RAM) model, the sequence $E$ simply takes $O(|E|)$ time, so setting $\ell$ to a constant function captures the RAM model. As we show later, by setting $\ell$ to a logarithmic function, we can model the cost of the TLB in virtual memory translation.

The goal of this paper is not to define locality functions for various models of computation. Instead, our results show that cache-oblivious algorithms go beyond minimizing the number of cache misses. In particular, we show that optimal cache-oblivious algorithms are locality-of-reference optimal, meaning, they are asymptotically optimal with respect to any choice of locality function, subject to some mild constraints (needed to ensure that these functions reward locality of reference). In Section 3 we present our result in a simplified setting, in which we focus on the algorithms that do not benefit from large cache sizes. In Section 4 we generalize the result to more general algorithms that do utilize the full cache.

1.2 Example application: optimality of van Emde Boas layout.

We now demonstrate an example application of our results which will appear as Theorem 3.1: namely, that the van Emde Boas layout—a layout for implicit static search trees that is optimal for cache-oblivious searching [9]—is also optimal for address translation on modern virtual memory architectures. This cost of address translation is non-negligible and has been observed to impact the performance of fundamental algorithms such as sorting and permuting in practice [10].

Let us review modern virtual memory design. Consider a machine that uses $U$ bits for addressing memory. Virtual memory is implemented as a trie of degree $2^{b}$, for some parameter $b$, where the translation process translates $b$ bits at a time, starting from the most significant bits of the address. In the worst-case, one translation necessitates $U/b$ lookups, but this cost is often much lower when TLB caching is used. In particular, if two addresses $e_{i}$ and $e_{i-1}$ share $kb$ most significant bits, then after translating the first memory address, the first $k$ steps of the second translation are cached in the TLB. As a result, the cost function associated with virtual memory translation is essentially $\ell(|e_{i}-e_{i-1}|)=\log_{b}(|e_{i}-e_{i-1}|)$. This function is clearly non-negative, non-decreasing and concave, thus, satisfying the requirements of Theorem 3.1.

The van Emde Boas layout of an implicit complete binary search tree (for brevity we’ll call it vEB tree) is defined as follows. Given a complete binary search tree on $n$ vertices of height $h=\lceil\log n\rceil$, let $T_{0}$ be the top subtree of height $\lceil h/2\rceil$ and let $T_{1},\dots,T_{\sqrt{n}}$ be the subtrees rooted at the children of the leaves of $T_{0}$. Then, vEB tree is defined by placing the subtree $T_{0}$ in a contiguous portion of an array, immediately followed by $T_{1},T_{2},\dots,T_{\sqrt{n}}$, with each subtree $T_{i}$ laid out recursively. Search on vEB tree is known to incur $O(\log_{B}n)$ cache misses, where $B$ is the block size of the DAM model [9].

Consider a machine that is equipped with virtual memory with parameters $U=O(\log n)$ and $b=O(1)$. The root-to-leaf traversal of vEB tree recursively traverses $T_{0}$, jumps by at most $n-\sqrt{n}$ addresses from a leaf of $T_{0}$ to the root of the appropriate subtree $T_{i}$, and recursively traverses $T_{i}$. Thus, the cost $Q^{\textsc{LoR}}_{\ell}(n)$ of the traversal using the above locality of reference function $\ell$ for virtual memory can be computed using the following recurrence:

[TABLE]

This solves to $Q^{\textsc{LoR}}_{\ell}(n)=O(\log n\log\log n)$. And Theorem 3.1 implies that this is an asymptotically optimal LoR cost for virtual address translation for the above function $\ell$.

1.3 Related work.

The closest work to our presentation here is the Hierarchical Memory Model [1]. In this model, accessing memory location $x$ takes time $f(x)$. This was extended to a blocked version where accessing $k$ memory locations takes times $f(x)+k$ [2]. In particular the case where $f(x)=\log x$ was studied and optimality obtained for a number of problems. This model, through its use of the memory cost function $f$, bears a number of similarities to ours, and it is meant to represent a multi-level cache where the user manually controls the movement of data from slow to fast memory. However, while it is able to capture temporal coherence well, even in the blocked version it does not capture fully the idea of spatio-temporal locality of reference, where an access is fast because it was close to something accessed recently.

Another model that proposed analyzing algorithm performance on a multi-level memory hierarchy is the Uniform Memory Hierarchy model (UMH) [4]. The UMH model is a multi-level variation of the DAM that simplifies analysis by assuming that the block size increases by a fixed factor at each level and that there is a uniform ratio between block and memory size at each level of the hierarchy. Unfortunately, this assumption is quite restrictive and doesn’t hold in practice on modern hardware.

2 Preliminaries

2.1 Models of computation

Let $P$ be a problem, $\mathcal{I}^{P}=\{I_{1},I_{2},\ldots\}$ be a set of valid instances (input sequences) for which problem $P$ can be solved, and $\mathcal{I}_{n}^{P}=\{I\in P:|I|=n\}$ be the subset of instances of $P$ with input of size $n$. Let $E(A,I)=\{e_{1},e_{2},\ldots\}$ be the sequence of accesses (reads and writes) to memory locations that arises by executing algorithm $A$ on instance $I$, and let $\mathcal{A}_{P}=\{A_{1},A_{2},\ldots\}$ denote the set of all algorithms that correctly solve $P$, i.e., generate a correct output for every instance in $\mathcal{I}^{P}$.

The general ideal-cache and LRU-cache models incorporate memory size $M$ and block size $B$ when computing the cost of an execution sequence. The $\frac{M}{B}$ blocks stored in internal memory make up the working set, and we define $\mathcal{W}^{\mathbb{M}}_{M,B}(E,i)$ to be the working set after the $i$-th access to the execution sequence $E$ in model $\mathbb{M}$. In the ideal-cache model (and the DAM model) the evictions from the working set $\mathcal{W}^{\textsc{opt}}_{M,B}$ are selected such that the total cost of executing $E$ is minimized [9], while in the LRU-cache model the evictions from the working set $\mathcal{W}^{\textsc{lru}}_{M,B}$ follow the least recently used policy [13]. Let $Q^{\mathbb{M}}_{x}(E(A,I))$ denote the cost of executing algorithm $A$ on instance $I$ in model $\mathbb{M}$ with model parameters $x$. Then the cost of algorithm $A$ on instance $I$ in the ideal-cache or LRU-cache models with cache parameters $M$ and $B$ is

[TABLE]

for $\mathbb{M}\in\{\textsc{co},\textsc{lru}\}.$ If $E(A,I)$ only depends on the input instance $I$ and not on any of the parameters of the model (e.g., $M$ or $B$) then the algorithm is cache-oblivious. Effectively, a cache-oblivious algorithm is one that runs in the classical RAM (with one level of memory). On the other hand, if the algorithm explicitly uses multiple levels of memory, or if the access pattern $E(A,I)$ depends on the particular values of $M$ or $B$, then it is called cache-aware. A more rigorous and formal definition of the working set, cache replacement policies, and the cache-oblivious and LRU costs is included in the full version of the paper.In Sections 3 and 4 we also define the cost $Q^{\textsc{LoR}}_{\ell}(E(A,I))$ of an algorithm on instance $I$ in our memoryless and general Locality of Reference (LoR) models with locality function $\ell$.

Similarly, we define $W^{\mathbb{M}}_{x}(P,A,n)=\max_{I\in\mathcal{I}_{n}^{P}}\{Q^{\mathbb{M}}_{x}(E(A,I)\}$ as the worst-case cost of algorithm $A$ on problem instances of size $n$ for problem $P$ on model $\mathbb{M}$ with parameters $x$.

Analysis of both cache-oblivious and cache-aware algorithms relies on additional constraints defining the relationship between the cache parameters $M$ and $B$. For example, the analysis of the cache-aware sorting algorithms [3] assumes $M\geq 2B$, while the analysis of the cache-oblivious sorting algorithms [9] and cache-oblivious sparse matrix dense vector (SpMV) multiplication [6] assume, respectively, $M\geq B^{2}$ and $M\geq B^{1+\epsilon}$ – the so-called tall-cache assumption. Therefore, let $\mathcal{M}_{P}(B)$ denote the set of all values of $M$ (as a function of $B$) that satisfy such constraints for problem $P$.

To prove our results, we need to rigorously define algorithm optimality. Otherwise, as there are multiple parameters involved, the order of the quantifiers would be unclear and ambiguous.

Definition 2.1

A cache-oblivious algorithm $A$ for problem $P$ is asymptotically CO-optimal in the ideal-cache model with the cache parameters $B$ and $M\in\mathcal{M}_{P}(B)$ iff111A standard and understandable reaction to this definition and the math to come is to sneer at the chain of quantifiers and declare “just write this using asymptotic notation!” Unfortunately, such notation is not flexible enough to say exactly what we need to say, and thus we make the quantification explicit. :

[TABLE]

To implement the cache replacement policy of the ideal-cache model requires the knowledge of what an algorithm will do in the future. Instead, modern hardware caches implement an approximation of the LRU-cache model, each time evicting the block that has been accessed least recently. While it is often easier to analyze the cache misses of an algorithm in the ideal-cache model, in this work we are able to work directly in the LRU-cache model.

Definition 2.2

A cache-oblivious algorithm $A$ for problem $P$ is asymptotically LRU-optimal in the LRU-cache model with the cache parameters $B$ and $M\in\mathcal{M}_{P}(B)$ iff:

[TABLE]

For completeness, however, we also prove our results in the ideal-cache model, by utilizing the following well-known resource-augmentation result of Sleator and Tarjan [12] (which also applies to other reasonable cache-replacement policies):

Lemma 2.1

[12]** Any LRU-optimal algorithm in the LRU-cache model with cache parameters $B$ and $M$ is CO-optimal in the ideal-cache model with cache parameters $B$ and $2M$.

The equivalence in optimality between the ideal-cache and LRU-cache models relies on the cache augmentation, and says nothing about asymptotic equivalence for the same $M$. However, this is not an issue for a large class of natural problems, which can be solved using memory-smooth cache-oblivious algorithms.

Definition 2.3

A cache-oblivious algorithm $A$ is memory-smooth iff increasing the memory size $M$ by a constant factor does not asymptotically change its execution cost. That is,

[TABLE]

where the $\Theta$-notation is with respect to the size of instance $I$.

Finally, we define algorithm optimality in our LoR model:

Definition 2.4

Let $\mathcal{L}$ be a class of functions. Algorithm $A$ for problem $P$ is asymptotically LoR-optimal with respect to $\mathcal{L}$ iff:

[TABLE]

The notion of an LoR-optimal algorithm with respect to all possible functions would be very powerful, as such an algorithm would be asymptotically optimal on any computing device that rewards locality of reference. In this paper we come very close to achieving such optimality, requiring only a natural set of restrictions on the functions in $\mathcal{L}$.

2.2 $B$-stable problems

To show the equivalence between LoR-optimal and CO-optimal algorithms, we must avoid pathological problems with worst-case behavior that varies dramatically with different instances of the problem for different block sizes.

We say that a problem is $B$-stable if, for any algorithm $A$ that solves $P$, there is some “worst-case” instance $I_{w}\in\mathcal{I}_{n}^{P}$ that, for every $B$, has CO cost asymptotically no less than the optimal worst-case cost for that $B$, over all instances. Formally,

Definition 2.5

Problem $P$ is $B$-stable if, for any algorithm $A\in\mathcal{A}_{P}$ that solves $P$:

[TABLE]

Intuitively, for any algorithm that solves a $B$-stable problem, there must be a single instance that, for all block sizes, has cost no less than the asymptotically worst-case optimal cost.

The following lemma, implied by the definition of CO-optimality (Definition 2.1), states that every algorithm must have an instance on which it performs no better, asymptotically, than the CO-optimal algorithm, for every $B$:

Lemma 2.2

If an asymptotically CO-optimal algorithm $A$ solves a $B$-stable problem $P$, then

[TABLE]

In the full version of the paper we prove the following lemma, which shows existence of non-$B$-stable problems, for which our main result (Theorem 3.1) does not hold. This justifies our classification and exclusion of these pathological cases.

Lemma 2.3

There exists a problem $P$ which is not $B$-stable and which has a CO-optimal algorithm which is not LoR-optimal.

2.3 $B$-smoothed analysis

To prove our results, we will work with the smoothed version of the analysis. Given the access sequence $E=E(A,I)$ and an integer $s$ chosen uniformly at random in the range $[0,B)$, let $E_{B\text{-smooth}}$ be the sequence derived from $E$ where every element of $E$ is increased by $s$. Then, the $B$-smoothed costs are defined as the expected cost on the $B$-smooth sequence in the respective cache models, i.e., ${Q^{\textsc{sco}}_{M,B}}(E)=\operatorname{\mathbb{E}}\left[Q^{\textsc{co}}_{M,B}(E_{B\text{-smooth}}\right]$ and ${Q^{\textsc{slru}}_{M,B}}(E)=\operatorname{\mathbb{E}}\left[{Q^{\textsc{lru}}_{M,B}}(E_{B\text{-smooth}})\right]$.

Lemma 2.4

For any execution sequence $E$

[TABLE]

and

[TABLE]

• Proof.

Shifting the execution sequence may cause accesses that were in the same block to become in two neighboring blocks, and accesses that were in two neighboring blocks to become in the same block. Thus, the cost may grow or shrink by a factor of two, but not more.

3 Memoryless algorithms

We begin with the simplest case, namely, the memoryless cache model (MCM) where the internal memory is just a single block, i.e. $\mathcal{M}_{P}(B)=\{B\}$.222While this might seem overly restrictive, a number of query algorithms on cache-oblivious data structures are CO-optimal, because one is always traversing further into the structure, and never comes back to revisit parts of memory near where it has already gone. Note that in this case, there is no need to differentiate between LRU-cache model and ideal-cache model because the working sets in both cache models, after accessing $e_{i}\in E(A,I)$, consist of a single block that contains $e_{i}$. Thus, the costs of the algorithm $A$ on instance $I$ in the MCM becomes

[TABLE]

This cost rewards spatial locality. Hence, in the LoR model it is natural to define the locality function $\ell$ to measure the cost of executing the sequence $E(A,I)$ as a function of the spatial distance $|e_{i}-e_{i-1}|$ between accesses:

[TABLE]

Let $\mathcal{L}$ denote the set of all non-negative, non-decreasing, concave functions $\ell:\mathbb{N}\rightarrow\mathbb{R}$. Even though $\mathcal{L}$ encompasses a wide range of (arbitrarily complicated) functions, we will show that any cache-oblivious algorithm is LoR-optimal with respect to $\mathcal{L}$ if and only if it is CO-optimal in the MCM.

3.1 Main result.

Let $\ell_{B}(d)=\min\left(1,\frac{d}{B}\right)$. We begin by proving our result for this specific locality function $\ell_{B}\in\mathcal{L}$ and generalize it to any $\ell\in\mathcal{L}$ later.

Lemma 3.1

For any execution sequence $E$ and block size $B$, ${Q^{\textsc{sco}}_{B,B}}(E)={Q^{\textsc{LoR}}_{\ell_{B}}}(E)$.

• Proof.

Consider a single access $e_{i}\in E$. Let $d=|e_{i}-e_{i-1}|$. If $d\geq B$ then $\ell_{B}(d)=1$, and the $i$th term of $Q^{\textsc{co}}_{B,B}(E)$ is also 1 because $\lfloor\frac{e_{i}+s}{B}\rfloor\neq\lfloor\frac{e_{i-1}+s}{B}\rfloor$ for all $0\leq s<B$. If $d<B$ then $\ell_{B}(d)=\frac{d}{B}$, and the expected value of the $i$th term of $Q^{\textsc{co}}_{B,B}(E)$ is also $\frac{d}{B}$, because $\lfloor\frac{e_{i}+s}{B}\rfloor\neq\lfloor\frac{e_{i-1}+s}{B}\rfloor$ for $d$ of the possible shifts $s\in[0,B)$.

Corollary 3.1

If a cache-oblivious algorithm $A$ for problem $P$ is asymptotically LoR-optimal with respect to $\mathcal{L}$, then it is asymptotically CO-optimal in the MCM.

• Proof.

Since $A$ is LoR-optimal, then it is within a constant factor of optimal for all locality functions, including $\ell_{B}(d)=\min\left(1,\frac{d}{B}\right)$. The corollary follows from Lemmas 3.1 and 2.4.

We now show that any locality function $\ell\in\mathcal{L}$ can be represented as a linear combination of $N$ functions $\ell_{B}(d)$, for $N$ various values of $B$.

{lem}

[]

For every locality function $\ell\in\mathcal{L}$ there exist non-negative constants $\alpha_{1},\alpha_{2}\ldots\alpha_{N}$ and $\beta_{1},\beta_{2}\ldots\beta_{N}$ such that $\ell(d)=\sum_{i=1}^{N}\alpha_{i}\ell_{\beta_{i}}(d)$ for integers $d$ in $[1..N]$.

• Proof.

[Proof (Sketch)] Let $\gamma_{i}=\begin{cases}2\ell(i)-\ell(i+1)&\text{ if }i=1\\ 2\ell(i)-\ell(i+1)-\ell(i-1)&\text{ if }2\leq i\leq N-1\\ \ell(i)-\ell(i-1)&\text{ if }i=N\end{cases}$ and let $\alpha_{i}=i\gamma_{i}$ and $\beta_{i}=i$. Then for any integer $x\in[1..N]$: $\sum_{i=1}^{N}\alpha_{i}\ell_{\beta_{i}}(x)=\sum_{i=1}^{N}\min\left(\alpha_{i},\frac{\alpha_{i}x}{\beta_{i}}\right)=\sum_{i=1}^{N}\min(i\gamma_{i},\gamma_{i}x)=\sum_{i=1}^{x-1}i\gamma_{i}+\gamma_{x}x+\sum_{i=x+1}^{N}\gamma_{i}x$ and this expression telescopes to $l(x)$ (the full derivation can be found in the full version of the paper.

Since function $\ell$ is non-negative and concave, all values of $\alpha_{i}$ and $\beta_{i}$ are non-negative.

{coro}

[]

For every locality function $\ell\in\mathcal{L}$ there exists a sets of $n$ non-negative constants $\alpha_{1},\alpha_{2}\ldots\alpha_{n}$, and $\beta_{1},\beta_{2}\ldots\beta_{n}$ such that, for any execution sequence $E$, $\sum_{i=1}^{n}\alpha_{i}{Q^{\textsc{LoR}}_{\ell_{\beta_{i}}}}(E)={Q^{\textsc{LoR}}_{\ell}}(E).$

Theorem 3.1

Let $\mathcal{L}$ be a set of all non-negative, non-decreasing, concave functions $l:\mathbb{N}\rightarrow\mathbb{R}$. Any cache-oblivious algorithm $A$ that solves a $B$-stable problem $P$ is LoR-optimal with respect to $\mathcal{L}$ if and only if it is CO-optimal in the memoryless cache model.

• Proof.

The first direction follows from Corollary 3.1. To prove the other direction, consider the CO-optimal algorithm $A_{\text{CO-Opt}}$ and some algorithm $A$ that solves $P$. Since $P$ is $B$-stable, by Definition 2.5,

[TABLE]

Thus, by Definition 2.4, $A_{\text{CO-Opt}}$ is asymptotically LoR-optimal.

4 General models for algorithms with memory

In the previous section, we considered execution sequences that did not utilize more than one block of memory, and thus locality to other than the previously accessed memory location was irrelevant. Now we generalize the model and apply it to execution sequences without this restriction. This requires that we consider the size and contents of internal memory when computing the expected cost of an access.

4.1 General LoR model

To capture the concept of the working set for algorithms that use internal memory, we define bidimensional locality functions that compute LoR cost based on two dimensions: distance and time. This bidimensional locality function, $\ell(d,\delta)$ represents the cost of a jump from a source element, $s$, to a target element, $t$, where $d$ and $\delta$ are the spatial distance and temporal distance, respectively, between $s$ and $t$. This captures the concept of the working set by using “time” to determine if the source element is in memory or not. If the source is temporally close (was accessed recently) and spatially close to the target, $t$, the resulting locality cost of the jump is small.

Details of the bidimensional locality functions. Let $\mathcal{L_{B}}$ denote the set of bidimensional locality functions that we consider. The functions in this set have the following properties. An element of this set is a function of the form $\ell(d,\delta)=\max(f(d),g(\delta))$, where $f(d)$ is non-negative, non-decreasing, and concave, while $g(\delta)$ is a 0-1 threshold function, i.e.,

[TABLE]

for some value $x$. For any $k\leq i$, the bidimensional locality cost of a jump from source element $e_{k}$ to target $e_{i}$ in the sequence $E$ is $\ell(|e_{i}-e_{k}|,t(E,i)-t(E,k))$, where $t(E,i)$ is the time of the $i$-th access. For simplicity of notation, we define $\delta_{t}(E,k,i)$ to be the temporal distance between the $i$-th access ($e_{i}$) and $k$-th access ($e_{k}$), i.e., $\delta_{t}(E,k,i)=t(E,i)-t(E,k)$. Intuitively, we can think of $\delta_{t}(E,k,i)$ as the time from access $e_{k}$ to “present” when accessing $e_{i}$. In addition, we require that the functions cannot be more “sensitive” to temporal locality than spatial locality, i.e., for any locality function $\ell(d,\delta)=\max(f(d),g(\delta))$, we have that $\forall_{x}[f(x)\geq g(x)]$. This corresponds to the tall cache assumption $M\geq B^{2}$, which is typically used in the analysis of cache-oblivious algorithms [7, 11]. Therefore, we restrict the machine parameters, $M$ and $B$, to all values $B\geq 1$ and $\mathcal{M}_{P}(B)=\{M:M\geq B^{2}\}$. A more in-depth discussion of the tall cache assumption and how it relates to the LoR model can be found in the vull version of the paper.

We form our definition of time based on the amount of change that occurs to the working set. For example, if an access causes a block of $B$ elements to be evicted, we say that time increases by 1. Thus, time depends on the locality function, so we define the time of the $i$-th access of $E$, for the given locality function $\ell$ to be $t_{\ell}(E,i)=\sum_{k=1}^{i-1}({Q^{\textsc{LoR}}_{\ell}}(E,k))$. That is, the time of access $e_{i}\in E$ is simply the sum of costs of all accesses prior to $e_{i}$ in sequence $E$. We note that the time after the last access of $E$ is the total LoR cost (i.e., ${Q^{\textsc{LoR}}_{\ell}}(E)=t_{\ell}(E,|E|+1)$).

Unlike the memoryless LoR cost, we cannot simply compute the cost of access $e_{i}$ using the distance from the previous access, $e_{i-1}$, since any of the prior accesses may be in the working set when accessing $e_{i}$. Furthermore, since we no longer consider only non-decreasing execution sequences, when accessing $e_{i}$, there may be accesses to both the left and right that could be in the same block as $e_{i}$. Therefore, computing ${Q^{\textsc{LoR}}_{\ell}}(E,i)$ using the locality function from a single source is insufficient to capture the idea of the working set, and a detailed example showing why this is the case is included in the full version of the paper. We define the general LoR cost of access $e_{i}\in E$ as ${Q^{\textsc{LoR}}_{\ell}}(E,i)=$

[TABLE]

Intuitively, the LoR cost of access $e_{i}\in E$ is computed from the minimum cost jumps from both the left side and right side of $e_{i}$. We note that this generalizes the LoR cost definition of the memoryless setting (Section 3), as the locality function from source $e_{R}$ will always evaluate to 1 for non-decreasing accesses.

This formulation has the added benefit that it lets us easily visualize an execution sequence in a graphical representation, illustrated in Figure 2. We consider a series of accesses in execution sequence $E$ as points in a 2-dimensional plane. The point representing access $e_{i}$ is plotted with the $x$ and $y$ coordinates corresponding to the spatial position, $e_{i}$, and the temporal position, $t(E,i)$, respectively. The cost of access $e_{i}$ is simply computed from the LoR cost with sources $e_{L}$ and $e_{R}$ (the previous access with the minimum locality function cost to the left and right, respectively). We can visually determine which previous accesses correspond to $e_{L}$ and $e_{R}$: if a previous access is outside the gray region (i.e., $\delta>\frac{M}{B}$ or $d>B$), the cost is 1. Otherwise, it is simply $\frac{d}{B}$.

4.2 Equivalence to cache-oblivious cost

As with the memoryless LoR model, we first prove our result for a specific bidimensional locality function, $\ell_{M,B}(d,\delta)~{}=~{}\max\left(\min\left(1,\frac{d}{B}\right),\min\left(1,\left\lfloor\frac{\delta}{M/B}\right\rfloor\right)\right)$, and later generalize our result to all bidimensional locality function in $\mathcal{L_{B}}$.

{thrm}

[]

For any cache-oblivious algorithm $A$, $B\geq 1$, $M\geq B^{2}$, execution sequence $E$ generated by $A$, and $\ell_{M,B}(d,\delta)~{}=~{}\max\left(\min\left(1,\frac{d}{B}\right),\min\left(1,\left\lfloor\frac{\delta}{M/B}\right\rfloor\right)\right)$: $Q^{\textsc{LoR}}_{{}_{\ell_{M,B}}}(E)={Q^{\textsc{slru}}_{M,B}}(E)$ and, consequently, ${Q^{\textsc{sco}}_{M,B}}(E)\leq Q^{\textsc{LoR}}_{{}_{\ell_{M,B}}}(E)\leq 2\cdot{Q^{\textsc{sco}}_{\frac{M}{2},B}}(E)$.

• Proof.

To prove that $Q^{\textsc{LoR}}_{{}_{\ell_{M,B}}}(E)={Q^{\textsc{slru}}_{M,B}}(E)$, we consider the cost of performing access $e_{i}\in E$. Assume that, when accessing $e_{i}$, $e_{L}$ is the nearest element to the left of $e_{i}$ ($e_{L}\leq e_{i}$) that is in the working set, i.e., $L<i$, $e_{L}\in\mathcal{W}^{\textsc{lru}}_{{}^{M,B}}(E,i-1)$, and $e_{i}-e_{L}$ is minimized. If there is no such element to the left of $e_{i}$ in the working set, then we say that $e_{L}=-\infty$. Similarly, assume that $e_{R}$ is the nearest element in $\mathcal{W}^{\textsc{lru}}_{{}^{M,B}}(E,i-1)$ to the right of $e_{i}$ ($e_{i}\leq e_{R}$), and if there is no such element, then $e_{R}=+\infty$. By this definition, for any access $e_{i}$, we can simply consider the spatial components, because, if no element is within temporal distance $\frac{M}{B}$, the spatial distance is $\infty$ and the $\ell_{M,B}$ cost is 1.

We consider three possible cases for the spatial distance of $e_{L}$ and $e_{R}$ from access $e_{i}$:

Case 1: $(e_{i}-e_{L})\geq B$ AND $(e_{R}-e_{i})\geq B$

There is no element in the working set within distance $B$ of $e_{i}$, then, for all alignment shifts, $0\leq s<B$, we know that $e_{i}\not\in\mathcal{W}^{\textsc{lru}}_{M,B}(E,i-1)$. Thus,

[TABLE]

We note that this includes the cases where $e_{L}$ and/or $e_{R}$ do not exist, since, we set $e_{L}=-\infty$ and/or $e_{R}=\infty$, respectively, in such cases.

Case 2: $(e_{i}-e_{L})<B$ OR $(e_{R}-e_{i})<B$

Only one side (left or right) is within distance $B$ of $e_{i}$. W.l.o.g, assume that $(e_{R}-e_{i})<B$ and $(e_{i}-e_{L})\geq B$. Since $(e_{i}-e_{L})\geq B$, for all shifts $0\leq s<B-1$, we know that $e_{L}\not\in\mathcal{W}^{\textsc{lru}}_{M,B}(E,i-1)$. Thus, the smoothed LRU cost is simply

[TABLE]

A symmetric argument holds in the case where $(e_{i}-e_{L})<B$ and $(e_{R}-e_{i})\geq B$.

Case 3: $(e_{i}-e_{L})<B$ AND $(e_{R}-e_{i})<B$

Both $e_{L}$ and $e_{R}$ are within distance $B$ of $e_{i}$, so the smoothed LRU cost depends on the number of alignment shifts, $s$, for which $e_{i}$ is not in the same block as either $e_{L}$ or $e_{R}$, i.e.,

[TABLE]

where

[TABLE]

Since they are equivalent for any access, $e_{i}\in E$, then for any execution sequence $E$,

[TABLE]

Since the cache-oblivious cost is computed assuming ideal cache replacement, and LRU cache replacement with twice the memory is 2-competitive with ideal cache, we have

[TABLE]

We can also prove similar asymptotic equivalence result between the LoR and the ideal-cache models for the same $M$, if we consider memory-smooth algorithms:

Lemma 4.1

For any memory-smooth cache-oblivious algorithm $A$, $B\geq 1$, $M\geq B^{2}$, execution sequence $E$ generated by $A$, and $\ell_{M,B}(d,\delta)~{}=~{}\max\left(\min\left(1,\frac{d}{B}\right),\min\left(1,\left\lfloor\frac{\delta}{M/B}\right\rfloor\right)\right)$:

[TABLE]

• Proof.

If $A$ is memory-smooth, then ${Q^{\textsc{sco}}_{M,B}}(E)=\Theta({Q^{\textsc{sco}}_{\frac{M}{2},B}}(E))$, and, by Theorem 4.2, $Q^{\textsc{LoR}}_{{}_{\ell_{M,B}}}(E)=\Theta\left({Q^{\textsc{sco}}_{M,B}}(E)\right)$.

4.3 Main result

We now extend our result to any bidimensional locality function $\ell\in\mathcal{L_{B}}$.

{thrm}[]

A cache-oblivious algorithm $A$ for a $B$-stable problem $P$ is LRU-optimal if and only if it is LoR-optimal with respect to $\mathcal{L_{B}}$, where $\mathcal{L_{B}}$ is a set of all functions of the form $\ell(d,\delta)=\max(f(d),g(\delta))$, where $g(\delta)$ is a 0-1 threshold function, $f(x)\geq g(x)$ for all $x\geq 0$, and $f$ is a non-negative, non-decreasing, concave function.

• Proof.

If algorithm $A_{\text{LoR}}$ is LoR-optimal for all bidimensional locality functions, then it is optimal for locality functions $\ell_{M,B}$, for any $M$ and $B$. By Theorem 4.2, it follows that $A_{\text{LoR}}$ is LRU-optimal for any $M$ and $B$.

[TABLE]

where $B\geq 1$ and $M\geq B^{2}$. Consider an arbitrary bidimensional locality function $\ell(d,\delta)=\max(f(d),g(\delta))$. By Lemma 3.1, we can represent the $f(d)$ component by a linear combination of $n$ memoryless $\ell_{B}$ functions (and therefore using the spatial component of $\ell_{M,B}$ functions). By our definition of bidimensional locality functions, $g(\delta)=\left\lfloor\frac{\delta}{x}\right\rfloor$, for some integer $x$. Thus, we simply set the temporal component of every one of our $\ell_{M,B}$ functions to be $g(\delta)$. For a given bidimensional locality function $\ell$, we define $\ell^{\ell}_{k}$ to be the $k$-th such $\ell_{M,B}$ function that we use to represent it, i.e., $\ell^{\ell}_{k}=\max(\alpha^{\ell}_{k}\ell_{\beta^{\ell}_{k}},g(\delta))$.333We note that, because we are limited to $M\geq B^{2}$ for our $\ell_{M,B}$ functions, we can only construct functions where $f(k)\geq g(k)$, for all $k>0$. However, our definition of bidimensional locality functions includes this restriction, as it corresponds to the tall cache assumption (discussed in Section 4.1).

Thus, we have

[TABLE]

Therefore, any LRU-optimal algorithm is also LoR-optimal.

{thrm}

[]

A memory-smooth cache-oblivious algorithm $A$ for a $B$-stable problem $P$ is CO-optimal if and only if $A$ is LoR-optimal with respect to $\mathcal{L_{B}}$, where $\mathcal{L_{B}}$ is a set of all functions of the form $\ell(d,\delta)=\max(f(d),g(\delta))$, where $g(\delta)$ is a 0-1 threshold function, $f(x)\geq g(x)$ for all $x\geq 0$, and $f$ is a non-negative, non-decreasing, concave function.

• Proof.

Since the cache-oblivious model assumes ideal cache replacement, for any execution sequence $E$, ${Q^{\textsc{sco}}_{M,B}}(E)\leq{Q^{\textsc{slru}}_{M,B}}(E)\leq 2\cdot{Q^{\textsc{sco}}_{\frac{M}{2},B}}(E)$. Since algorithm $A$ is memory-smooth, for any execution sequence $E$ generated by $A$, ${Q^{\textsc{sco}}_{\frac{M}{2},B}}(E)=\Theta({Q^{\textsc{sco}}_{M,B}}(E))$. Therefore, ${Q^{\textsc{slru}}_{M,B}}(E)=\Theta({Q^{\textsc{sco}}_{M,B}}(E))$. Since the LRU cost and CO cost are asymptotically equivalent for every execution sequence generated by $A$, then $A$ is asymptotically LRU-optimal if and only if it is asymptotically CO-optimal and, by Theorem 4.3, $A$ is LoR-optimal if and only if it is CO-optimal.

5 Conclusion

Despite the increasing complexity of modern hardware architectures, the goal of many design and optimization principles remain the same: maximize locality of reference. Even many of the optimization techniques used by modern compilers, such as branch prediction or loop unrolling [13], can be seen as methods of increasing spatial and/or temporal locality. As we demonstrated in this work, cache-oblivious algorithms do just that, suggesting that the performance benefits of such algorithms extend beyond what was originally envisioned.

That is to say, although we have introduced a new way to model computation via locality functions, we are not advocating algorithm design and analysis using locality functions. Instead, though our transformations, we have shown that creating the best possible algorithms in the existing cache-oblivious models is they right way to design algorithms not just for a multi-level cache, but for any locality-of-reference-rewarding system. One can thus conclude that the cache-oblivious model is better than we thought it was.

A Deferred proofs

A.1 Proof of Lemma 3.1

See 3.1

• Proof.

Let

[TABLE]

and let $\alpha_{i}=i\gamma_{i}$ and $\beta_{i}=i$. Since $\ell$ is a non-negative and concave, all $\gamma_{i}$ values are non-negative, and, consequently, all $\alpha_{i}$ and $\beta_{i}$ values are also non-negative.

Thus:

[TABLE]

B Formal definitions of cache-oblivious and LRU cost

Analysis of cache-oblivious algorithms assumes $M$ to be the size of internal memory, with $\frac{M}{B}$ blocks being stored in internal memory at a given time, which we call the working set. The working set is made up of blocks of contiguous memory, each containing $B$ elements. For a given block size, $B$, we enumerate the blocks of memory by defining the block containing element $e$ as $\{e\}_{B}$ (the $\left\lfloor\frac{e}{B}\right\rfloor$-th block). Formally, we define the working set after the $i$-th access of execution sequence $E$ on a system with memory size $M$, block size $B$, and cache replacement policy $\mathcal{P}$ (formally defined below) as $\mathcal{W}^{\mathcal{P}}_{M,B}(E,i)$. For simplicity of notation, we refer to the working set after the $i$-th access simply as $\mathcal{W}_{i}$ when the other parameters ($M$, $B$, $\mathcal{P}$, and $E$) are unambiguous.

When we access an element $e_{i}$, if the block containing $e_{i}$ is in the working set (i.e., $\{e_{i}\}_{B}\in\mathcal{W}_{i-1}$), it is a cache hit and, in the cache-oblivious model, it has a cost of 0. However, if $\{e_{i}\}_{B}$ is not in the working set, it is a cache miss, resulting in a cost of 1. On a cache miss, the accessed block, $\{e_{i}\}_{B}$ is loaded into memory, replacing an existing block, which is determined by cache replacement policy. We define a general cache replacement policy as a function that selects the block of the working set to evict when a cache miss occurs, i.e., for memory size $M$ and block size $B$:

[TABLE]

where $\mathcal{P}_{M,B}(E,\mathcal{W}_{i-1},i)$ defines the block to be evicted and $\{e_{i}\}_{B}$ is the new block being added to the working set. Since a cache miss results in a cost of 1 and a cache hit has cost 0, the total cost of execution sequence $E$ is simply:

[TABLE]

For this work, we focus on the following cache replacement policies:

• •

$\mathcal{P}^{\textsc{opt}}_{{}^{M,B}}(E,\mathcal{W},i)$: The ideal cache replacement policy with internal memory size $M$ and block size $B$. The number of evictions (and cache misses) over execution sequence $E$ is minimized. This is equivalent to Belady’s algorithm [5] that evicts the block $\{e_{k}\}_{B}$ that is accessed the farthest in the future among all blocks in $\mathcal{W}$.

• •

$\mathcal{P}^{\textsc{lru}}_{{}^{M,B}}(E,\mathcal{W},i)$: The least recently used (LRU) cache replacement policy with internal memory size $M$ and block size $B$. The evicted block, $\{e_{k}\}_{B}$, is the “least recently used” block in $\mathcal{W}$. That is, $\{e_{k}\}_{B}$ is selected such that no element in $\{e_{k}\}_{B}$ has been accessed more recently than the most recently accessed element of any other block in $\mathcal{W}$.

We define $\mathcal{W}^{\textsc{opt}}_{{}^{M,B}}(E,i)$ and $\mathcal{W}^{\textsc{lru}}_{{}^{M,B}}(E,i)$ as the working sets after the $i$-th access of sequence $E$, when using the ideal and LRU cache replacement policies, respectively. Thus, the cache-oblivious cost (using the ideal cache replacement policy) of performing the $i$-th access of on a system with memory size $M$ and block size $B$ is ${Q^{\textsc{co}}_{M,B}}(E,i)~{}=~{}\begin{cases}0\text{ if }\{e_{i}\}_{B}\in\mathcal{W}^{\textsc{opt}}_{{}^{M,B}}(E,i{-}1)\\ 1\text{ otherwise}\\ \end{cases}$ and the total cost for the entire execution sequence $E$ is ${Q^{\textsc{co}}_{M,B}}(E)=\sum_{i=1}^{|E|}{Q^{\textsc{co}}_{M,B}}(E,i)$. We similarly define the cost with the LRU cache replacement policy for a single access $e_{i}$ and a total execution sequence $E$ as ${Q^{\textsc{lru}}_{M,B}}(E,i)$ and ${Q^{\textsc{lru}}_{M,B}}(E)$, respectively.

Theorem B.1

For any execution sequence, $E$, memory size $M$, and block size $B$, the total cache misses using the LRU cache replacement policy with a memory twice the size ($2M$) is 2-competitive with number of cache misses using the ideal cache replacement policy, i.e., ${Q^{\textsc{lru}}_{2M,B}}(E)\leq 2\cdot{Q^{\textsc{co}}_{M,B}}(E)$.

• Proof.

It follows from the work of Sleator and Tarjan [12].

C On the tall cache assumption

Cache-oblivious algorithms are analyzed for memory size $M$ and block size $B$ and the tall cache assumption simply assumes that $M\geq B^{2}$. This assumption is required by many cache-obliviously optimal algorithms because many require that at least $B$ blocks can be loaded into internal memory at a time. It has been proven that without the tall cache assumption, one cannot achieve cache-oblivious optimality for several fundamental problems, including matrix transposition [11] and comparison-based sorting [7]. Thus, we consider how this assumption is reflected in the LoR model, and whether we can gain insight into the underlying need for the tall cache assumption.

Recall that our class of bidimensional locality functions are of the form $\ell(d,\delta)=\max(f(d),g(\delta))$, where $f$ is subadditive and $g$ is a 0-1 threshold function. In Section 4.2 we define the locality function that corresponds to a memory system with memory size $M$ and block size $B$ to be

[TABLE]

thus, for this function, $f(d)=\min\left(1,\frac{d}{B}\right)$ and $g(\delta)=\min\left(1,\left\lfloor\frac{\delta}{M/B}\right\rfloor\right)$. The tall cache assumption states that $M\geq B^{2}$, or $\frac{M}{B}>B$. This is reflected in our locality function as the requirement that $\forall_{k\geq 0}[f(k)\geq g(k)]$. This restriction between $f$ and $g$ implies that $\ell$ cannot be more “sensitive” to temporal locality than spatial locality. That is, the LoR cost when spatial and temporal distance are equal will be computed from the spatial distance (i.e., $\ell(d,\delta)=f(d)$ if $d\geq\delta$). Additionally, this implies that $\ell(x,x)$ is subadditive. Intuitively, this tells us that, with the tall cache assumption, any algorithm that balances spatial and temporal locality of reference will not have performance limited by temporal locality. Many cache-obliviously optimal algorithms aim to balance spatial and temporal locality, thus requiring the tall cache assumption to achieve optimality.

D A single LoR source does not represent the working set

In this section, we show that computing the general LoR cost using only a single source (with the minimum cost) is insufficient to represent the working set. Specifically, we show the potential discrepancy between such a formulation of LoR cost and the smooth LRU cost. Informally, we show two execution sequences where the sequence of distances from the closest previous accessed object is given by:

[TABLE]

but all the accesses in the first sequence lie within one block, while in the second, $\frac{1}{2}\log B$ blocks are accessed. This shows that the distance from the closest previous access by itself can not characterize the runtime.

We formally define this single-source definition of LoR cost of accessing $e_{i}$ as $\widehat{Q^{\textsc{LoR}}_{\ell}}(E,i)=\min_{k=1}^{i-1}\ell(|e_{i}-e_{k}|,\delta_{t}(E,k,i))$. To show the discrepancy between this formulation and the LRU cost, we consider the specific locality function that corresponds to the LRU cost for a specific memory size $M$ and block size $B$: $\ell_{M,B}(d,\delta)=\max\left(\min\left(1,\frac{d}{B}\right),\min\left(1,\left\lfloor\frac{\delta}{M/B}\right\rfloor\right)\right)$.

Given an array of elements, $a$, located in contiguous memory, we define $a[i]$ as the $i$-th element in array $a$. Consider an execution sequence $E$ that first accesses $a[0]$ and $a[B]$, then performs a series of stages of accesses of elements within the range $[0,B]$. At the first stage, $a[B/2]$ is accessed. At the second stage, $a[B/4]$ and $a[3B/4]$ are accessed. At stage 3, $a[B/8]$, $a[3B/8]$, $a[5B/8]$ and $a[7B/8]$ are accessed, and so on for $\log{B}$ stages. By tall cache, we know that $M>2B$, so at any stage $k$, the blocks containing elements accessed during the previous stage are in the working set. Thus, the LRU cost of execution sequence $E$ is ${Q^{\textsc{slru}}_{M,B}}(E)=$

[TABLE]

For every access, all elements element between $a[0]$ and $a[B]$ will always be in the working set, for every shift value $s$. Thus, the cost is 0 at each stage after the first two accesses, so ${Q^{\textsc{slru}}_{M,B}}(E)=2$.

The single-source LoR cost, $\widehat{Q^{\textsc{LoR}}_{\ell_{M,B}}}(E)$, however, depends only on the single access in the working set with the smallest spatial distance (i.e., minimum LoR cost). Since, at each stage, the accesses from the previous stage have temporal distance $<\frac{M}{B}$, the temporal component of the locality function is always 0 and $\frac{d}{B}$ dictates the cost of each access. At each stage, the spatial distance, $d$, decreases by a factor 2, thus

[TABLE]

Thus, the single-source cost formulation does not generalize the LRU cost, while using two sources does (as we prove in Lemma 4.2).

E Necessity of $B$ stability

The following lemma shows that Theorem 3.1 would not hold if the restriction to $B$-stable algorithms were to be removed.

Lemma E.1

There exists a problem $P$ which is not $B$-stable and which has a CO-optimal algorithm which is not LoR optimal.

• Proof.

Here we demonstrate a toy problem that meets the requirements of the lemma while also illustrating the unnaturalness of such problems. It has two candidate algorithms, one which has the same runtime on each instance, and a second one that for each instance has some values of $B$ that it runs faster than the first algorithm, and some that it runs more slowly than the first algorithm on, asymptotically. Thus for each $B$ the worst-case time of the first algorithm is better than the second, but there is no single bad instance for the second algorithm.

Consider a problem $P$ and a set $\mathcal{A}$ of two cache-oblivious algorithms $A_{1}$ and $A_{2}$. The problem, given an $n$, has a set of $n$ instances $\mathcal{I}_{n}=I^{n}_{1},I^{n}_{2},\ldots I^{n}_{n}$. The runtimes of the two algorithms are given as follows:

[TABLE]

These runtimes can be realized through an appropriately twisted problem definition that forces an algorithm for each instance to read all elements in one of two sets of memory locations in order to be considered a valid algorithm. In particular our problem admits two algorithms, one of which, $A_{2}$, can solve any instance by performing $n\log n\log\log\log n$ reads in memory generated by $n\log\log\log n$ searches in a van Emde Boas search structure, and the other, $A_{1}$, by reading at memory locations generated by an arithmetic progression, where the step and number of locations depends on the instance.

Accessing $k$ memory locations evenly spaced $\sigma$ apart takes $\Theta(1+\min\left(k,\frac{k\sigma}{B}\right))$ I/Os in the ideal-cache model; thus the desired runtime of algorithm $A_{1}$ on instance $I^{n}_{i}$ can be forced by having the algorithm $A_{1}$ instance $I_{i}$ read $\frac{n\log n\log\log n}{\log i}$ memory locations evenly spaced $i$ apart.

What are the worst-case runtimes of these algorithms?

[TABLE]

So, looking at these two algorithms, $A_{2}$ is clearly the worst-case optimal in the CO model.

Now, recall the definition of $B$-stability (Definition 2.5): Problem $P$ is $B$-stable if, for any algorithm $A$ that solves $P$,

[TABLE]

Applying this to our problem gives:

[TABLE]

This is false for all choices of ${I^{n}_{i}\in\mathcal{I}_{n}^{P}}$. Specifically, if $i\geq\log n$, then setting $B=2$ and using the first term of the min gives the following contradiction for any $c$ as $n$ grows:

[TABLE]

And, if $i\leq\log n$, then setting $B=n$ and using the second term on the right gives the following contradiction:

[TABLE]

This concludes the proof that $P$ is not $B$ stable. We now argue that while $A_{2}$ is CO-optimal for $P$, and $A_{1}$ is not, with locality function $\ell(d)=\log(d)$ the reverse is true as $A_{1}$ will have the asymptotically better runtime with this locality function in the LoR model.

In the introduction we mentioned that the LoR runtime with locality function $\ell(d)=\log d$ for searching in a vEB structure is $\Theta(\log n\log\log n)$, thus since $A_{2}$ does this $n\log\log\log n$ times its cost is $\Theta(n\log n\log\log n\log\log\log n)$. Algorithm $A_{1}$ is easy to analyze as on instance $I_{i}^{n}$ it accesses $\frac{n\log n\log\log n}{\log i}$ memory locations evenly spaced $i$ apart, thus its cost is

[TABLE]

Thus the CO-optimal $A_{2}$ has a LoR runtime (with $\ell(d)=\log d$) of $\Theta(\log n\log\log n\log\log\log n)$ which is a $\Theta(\log\log\log)$ factor worse than the non-CO-optimal $A_{1}$ with LoR runtime of $\log n\log\log n$. Since $A_{2}$ is not optimal for one locality function, it can not be optimal for all valid locality functions.

What made this problem not $B$-stable? It was the fact that every instance was constructed to be faster for one algorithm for some values of $B$ and slower for others than the optimal worst-case algorithm. In this example $A_{1}$ ran instance $I_{i}^{n}$ slower than $A_{2}$ for $B$ close to $i$ and faster than $A_{2}$ for $B$ far from $i$. However, it is far from natural to have an instance in effect encode faster-than-worse-case performance on selected $B$’s. In a standard data structure query, such as “what is the predecessor of a given item in an ordered set,” the query item itself has nothing that combined with the problem definition allows a query to encode a preference for fast execution for certain $B$’s in a non-optimal algorithm. We note that this is very different than some algorithms which may “hard-code” some instances and make them fast; this does not pose a problem with regards to $B$-stability as this makes this instance fast for all values of $B$.