Comparative Evaluation of Big-Data Systems on Scientific Image Analytics Workloads

TL;DR

This paper provides a comprehensive comparison of five big-data systems for scientific image analytics, revealing their strengths and weaknesses in supporting real-world use cases and highlighting areas for future improvement.

Contribution

It is the first study to evaluate large-scale image analysis systems on real scientific workloads, offering insights into their performance and usability challenges.

Findings

All evaluated systems have notable shortcomings.

Performance varies significantly across systems.

Opportunities exist to improve efficiency and usability.

Abstract

Scientific discoveries are increasingly driven by analyzing large volumes of image data. Many new libraries and specialized database management systems (DBMSs) have emerged to support such tasks. It is unclear, however, how well these systems support real-world image analysis use cases, and how performant are the image analytics tasks implemented on top of such systems. In this paper, we present the first comprehensive evaluation of large-scale image analysis systems using two real-world scientific image data processing use cases. We evaluate five representative systems (SciDB, Myria, Spark, Dask, and TensorFlow) and find that each of them has shortcomings that complicate implementation or hurt performance. Such shortcomings lead to new research opportunities in making large-scale image analysis both efficient and easy to use.