Pay One, Get Hundreds for Free: Reducing Cloud Costs through Shared Query Execution

TL;DR

This paper presents a query rewriting technique that combines multiple concurrent queries into a single shared execution plan, significantly reducing cloud query costs without system modifications.

Contribution

It introduces a novel query rewriting method for shared execution in cloud data analysis, lowering costs and increasing throughput without altering existing systems.

Findings

Shared execution reduces total work done compared to individual queries.

Cost savings of up to 100x in Amazon Athena and Google BigQuery.

Higher throughput achieved with shared query execution.

Abstract

Cloud-based data analysis is nowadays common practice because of the lower system management overhead as well as the pay-as-you-go pricing model. The pricing model, however, is not always suitable for query processing as heavy use results in high costs. For example, in query-as-a-service systems, where users are charged per processed byte, collections of queries accessing the same data frequently can become expensive. The problem is compounded by the limited options for the user to optimize query execution when using declarative interfaces such as SQL. In this paper, we show how, without modifying existing systems and without the involvement of the cloud provider, it is possible to significantly reduce the overhead, and hence the cost, of query-as-a-service systems. Our approach is based on query rewriting so that multiple concurrent queries are combined into a single query. Our experiments show the aggregated amount of work done by the shared execution is smaller than in a query-at-a-time approach. Since queries are charged per byte processed, the cost of executing a group of queries is often the same as executing a single one of them. As an example, we demonstrate how the shared execution of the TPC-H benchmark is up to 100x and 16x cheaper in Amazon Athena and Google BigQuery than using a query-at-a-time approach while achieving a higher throughput.