Array Requirements for Scientific Applications and an Implementation for Microsoft SQL Server

TL;DR

This paper discusses the design and implementation of an array data type extension for Microsoft SQL Server 2008, aimed at supporting high-performance scientific applications in fields like astrophysics and fluid dynamics.

Contribution

It introduces a comprehensive array extension for relational databases, including a working implementation and performance evaluation for scientific data processing.

Findings

Array extension improves data processing performance

Implementation supports large-scale scientific applications

Design lessons inform future database array support

Abstract

This paper outlines certain scenarios from the fields of astrophysics and fluid dynamics simulations which require high performance data warehouses that support array data type. A common feature of all these use cases is that subsetting and preprocessing the data on the server side (as far as possible inside the database server process) is necessary to avoid the client-server overhead and to minimize IO utilization. Analyzing and summarizing the requirements of the various fields help software engineers to come up with a comprehensive design of an array extension to relational database systems that covers a wide range of scientific applications. We also present a working implementation of an array data type for Microsoft SQL Server 2008 to support large-scale scientific applications. We introduce the design of the array type, results from a performance evaluation, and discuss the lessons learned from this implementation. The library can be downloaded from our website at http://voservices.net/sqlarray/