Data Management for High-Throughput Genomics

TL;DR

This paper investigates the use of relational database systems for managing and analyzing large-scale genomics data, focusing on storage, scalability, and performance challenges in high-throughput sequencing workflows.

Contribution

It presents a novel database design and prototype implementation for high-throughput genomics, exploring SQL-based data analysis and storage management within a relational database system.

Findings

Database design shows potential for managing large genomics datasets.

SQL-based analysis can be integrated into genomics workflows.

Initial results indicate scalability and performance considerations.

Abstract

Today's sequencing technology allows sequencing an individual genome within a few weeks for a fraction of the costs of the original Human Genome project. Genomics labs are faced with dozens of TB of data per week that have to be automatically processed and made available to scientists for further analysis. This paper explores the potential and the limitations of using relational database systems as the data processing platform for high-throughput genomics. In particular, we are interested in the storage management for high-throughput sequence data and in leveraging SQL and user-defined functions for data analysis inside a database system. We give an overview of a database design for high-throughput genomics, how we used a SQL Server database in some unconventional ways to prototype this scenario, and we will discuss some initial findings about the scalability and performance of such a more database-centric approach.