Upscaledb: Efficient Integer-Key Compression in a Key-Value Store using SIMD Instructions

TL;DR

This paper presents SIMD-accelerated integer key compression techniques for key-value stores, significantly reducing memory and improving query speed, demonstrated within the Upscaledb database system.

Contribution

It introduces novel SIMD-based algorithms for integer key compression that operate directly on compressed data, enhancing performance and compression ratio in database systems.

Findings

SIMD-accelerated binary packing achieves 40% faster queries than uncompressed data.

Compression reduces memory usage by up to a factor of ten.

Techniques are effective for both transactional and analytic workloads.

Abstract

Compression can sometimes improve performance by making more of the data available to the processors faster. We consider the compression of integer keys in a B+-tree index. For this purpose, systems such as IBM DB2 use variable-byte compression over differentially coded keys. We revisit this problem with various compression alternatives such as Google's VarIntGB, Binary Packing and Frame-of-Reference. In all cases, we describe algorithms that can operate directly on compressed data. Many of our alternatives exploit the single-instruction-multiple-data (SIMD) instructions supported by modern CPUs. We evaluate our techniques in a database environment provided by Upscaledb, a production-quality key-value database. Our best techniques are SIMD accelerated: they simultaneously reduce memory usage while improving single-threaded speeds. In particular, a differentially coded SIMD binary-packing techniques (BP128) can offer a superior query speed (e.g., 40% better than an uncompressed database) while providing the best compression (e.g., by a factor of ten). For analytic workloads, our fast compression techniques offer compelling benefits. Our software is available as open source.