A Fast Transformer-based General-Purpose Lossless Compressor

TL;DR

This paper introduces TRACE, a transformer-based lossless compressor that significantly reduces execution time while maintaining competitive compression ratios, by designing a lightweight, compression-friendly transformer structure and acceleration strategies.

Contribution

The paper proposes a novel, fast, general-purpose lossless compressor using a single-layer transformer with new model selection metrics and acceleration techniques, addressing computational inefficiency in existing methods.

Findings

TRACE achieves approximately 3x speedup over state-of-the-art compressors.

It maintains comparable compression ratios to existing methods.

The design enables efficient parallel history-dependency modeling in compression.

Abstract

Deep-learning-based compressor has received interests recently due to much improved compression ratio. However, modern approaches suffer from long execution time. To ease this problem, this paper targets on cutting down the execution time of deep-learning-based compressors. Building history-dependencies sequentially (e.g., recurrent neural networks) is responsible for long inference latency. Instead, we introduce transformer into deep learning compressors to build history-dependencies in parallel. However, existing transformer is too heavy in computation and incompatible to compression tasks. This paper proposes a fast general-purpose lossless compressor, TRACE, by designing a compression-friendly structure based on a single-layer transformer. We first design a new metric to advise the selection part of compression model structures. Byte-grouping and Shared-ffn schemes are further proposed to fully utilize the capacity of the single-layer transformer. These features allow TRACE to achieve competitive compression ratio and a much faster speed. In addition, we further accelerate the compression procedure by designing a controller to reduce the parameter updating overhead. Experiments show that TRACE achieves an overall $\sim$3x speedup while keeps a comparable compression ratio to the state-of-the-art compressors. The source code for TRACE and links to the datasets are available at https://github.com/mynotwo/A-Fast-Transformer-based-General-Purpose-LosslessCompressor.