Tight Lower Bounds for Query Processing on Streaming and External Memory Data

TL;DR

This paper establishes fundamental lower bounds for query processing in streaming and external memory models, demonstrating the limitations of scans, joins, sorting, and XPath evaluation under constrained resources.

Contribution

It introduces a strict hierarchy based on the number of data scans and proves tight bounds for query evaluation complexity in external memory models.

Findings

Hierarchy of data scans impacts processing feasibility

Joins and sorting are infeasible with limited memory and scans

Tight bounds for XPath evaluation and filtering

Abstract

We study a clean machine model for external memory and stream processing. We show that the number of scans of the external data induces a strict hierarchy (as long as work space is sufficiently small, e.g., polylogarithmic in the size of the input). We also show that neither joins nor sorting are feasible if the product of the number $r(n)$ of scans of the external memory and the size $s(n)$ of the internal memory buffers is sufficiently small, e.g., of size $o(\sqrt[5]{n})$. We also establish tight bounds for the complexity of XPath evaluation and filtering.