The Open Connectome Project Data Cluster: Scalable Analysis and Vision for High-Throughput Neuroscience

TL;DR

The paper presents a scalable, high-performance database system designed for analyzing high-throughput brain imaging data, enabling efficient construction of neural connectomes through distributed computing and RESTful interfaces.

Contribution

It introduces a novel distributed database architecture optimized for spatial brain imaging data, combining NoSQL principles with high-throughput computing for connectome analysis.

Findings

Achieved high throughput by spatial data partitioning.

Demonstrated effective use of parallel disk arrays and solid-state storage.

Validated system scalability and performance with real-world data.

Abstract

We describe a scalable database cluster for the spatial analysis and annotation of high-throughput brain imaging data, initially for 3-d electron microscopy image stacks, but for time-series and multi-channel data as well. The system was designed primarily for workloads that build connectomes---neural connectivity maps of the brain---using the parallel execution of computer vision algorithms on high-performance compute clusters. These services and open-science data sets are publicly available at http://openconnecto.me. The system design inherits much from NoSQL scale-out and data-intensive computing architectures. We distribute data to cluster nodes by partitioning a spatial index. We direct I/O to different systems---reads to parallel disk arrays and writes to solid-state storage---to avoid I/O interference and maximize throughput. All programming interfaces are RESTful Web services, which are simple and stateless, improving scalability and usability. We include a performance evaluation of the production system, highlighting the effectiveness of spatial data organization.