Achieving 100,000,000 database inserts per second using Accumulo and D4M

TL;DR

This paper demonstrates that by adapting supercomputing techniques, Accumulo can achieve over 100 million database inserts per second, significantly surpassing previous records and scaling linearly with system resources.

Contribution

It presents the first demonstration of achieving 100 million inserts per second in a database using Accumulo and D4M, applying supercomputing methods for high performance.

Findings

Achieved over 100 million inserts per second.

Performance scales linearly with clients, servers, and data size.

Utilized supercomputing techniques like distributed arrays and load balancing.

Abstract

The Apache Accumulo database is an open source relaxed consistency database that is widely used for government applications. Accumulo is designed to deliver high performance on unstructured data such as graphs of network data. This paper tests the performance of Accumulo using data from the Graph500 benchmark. The Dynamic Distributed Dimensional Data Model (D4M) software is used to implement the benchmark on a 216-node cluster running the MIT SuperCloud software stack. A peak performance of over 100,000,000 database inserts per second was achieved which is 100x larger than the highest previously published value for any other database. The performance scales linearly with the number of ingest clients, number of database servers, and data size. The performance was achieved by adapting several supercomputing techniques to this application: distributed arrays, domain decomposition, adaptive load balancing, and single-program-multiple-data programming.