A High-Throughput Hardware Accelerator for Lempel-Ziv 4 Compression Algorithm

TL;DR

This paper introduces a high-speed hardware architecture for the LZ4 compression algorithm that significantly improves throughput by restricting match length and parallelism, achieving over 16 Gb/s with minimal compression ratio loss.

Contribution

A novel hardware design that enhances LZ4 compression throughput by limiting match length and parallelism, surpassing previous implementations in speed.

Findings

Achieves up to 16.10 Gb/s throughput

Provides a 2.648x speed improvement over state-of-the-art

Maintains acceptable compression ratio reduction of 4.93% to 11.68%

Abstract

This paper delves into recent hardware implementations of the Lempel-Ziv 4 (LZ4) algorithm, highlighting two key factors that limit the throughput of single-kernel compressors. Firstly, the actual parallelism exhibited in single-kernel designs falls short of the theoretical potential. Secondly, the clock frequency is constrained due to the presence of the feedback loops. To tackle these challenges, we propose a novel scheme that restricts each parallelization window to a single match, thus elevating the level of actual parallelism. Furthermore, by restricting the maximum match length, we eliminate the feedback loops within the architecture, enabling a significant boost in throughput. Finally, we present a high-speed hardware architecture. The implementation results demonstrate that the proposed architecture achieves a throughput of up to 16.10 Gb/s, exhibiting a 2.648x improvement over the start-of-the-art. The new design only results in an acceptable compression ratio reduction ranging from 4.93% to 11.68% with various numbers of hash table entries, compared to the LZ4 compression ratio achieved by official software implementations disclosed on GitHub.