Efficient Iterative Processing in the SciDB Parallel Array Engine
Emad Soroush, Magdalena Balazinska, Simon Krughoff, Andrew Connolly
TL;DR
This paper introduces a model and optimizations for native iterative array processing in SciDB, significantly improving performance for scientific applications involving iterative computations on array data.
Contribution
It presents the first native support for iterative processing in an array database engine, with a comprehensive model and optimization techniques.
Findings
Enhanced performance on astronomy workloads
Effective optimization strategies for iterative array processing
Demonstrated benefits of native iterative support in SciDB
Abstract
Many scientific data-intensive applications perform iterative computations on array data. There exist multiple engines specialized for array processing. These engines efficiently support various types of operations, but none includes native support for iterative processing. In this paper, we develop a model for iterative array computations and a series of optimizations. We evaluate the benefits of an optimized, native support for iterative array processing on the SciDB engine and real workloads from the astronomy domain.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Data Storage Technologies · Distributed and Parallel Computing Systems · Astronomy and Astrophysical Research