A comparative analysis of state-of-the-art SQL-on-Hadoop systems for interactive analytics

TL;DR

This paper compares four leading SQL-on-Hadoop systems—Impala, Drill, Spark SQL, and Phoenix—evaluating their performance on benchmarks to understand their strengths, weaknesses, and execution characteristics in cloud environments.

Contribution

It provides a detailed comparative analysis of these systems' architectures, performance trade-offs, and query execution behaviors using standardized benchmarks.

Findings

Impala outperforms others in text format (4.41x - 6.65x)

Performance varies across systems and data formats

Execution profiles reveal bottlenecks and performance variations

Abstract

Hadoop is emerging as the primary data hub in enterprises, and SQL represents the de facto language for data analysis. This combination has led to the development of a variety of SQL-on-Hadoop systems in use today. While the various SQL-on-Hadoop systems target the same class of analytical workloads, their different architectures, design decisions and implementations impact query performance. In this work, we perform a comparative analysis of four state-of-the-art SQL-on-Hadoop systems (Impala, Drill, Spark SQL and Phoenix) using the Web Data Analytics micro benchmark and the TPC-H benchmark on the Amazon EC2 cloud platform. The TPC-H experiment results show that, although Impala outperforms other systems (4.41x - 6.65x) in the text format, trade-offs exists in the parquet format, with each system performing best on subsets of queries. A comprehensive analysis of execution profiles expands upon the performance results to provide insights into performance variations, performance bottlenecks and query execution characteristics.