Forward asymmetric numeral systems coding for natural language text compression

TL;DR

This paper introduces a method combining modeling and adaptive asymmetric numeral systems (ANS) coding to achieve fast, efficient compression of natural language text with near-entropy ratios.

Contribution

It proposes a novel approach that integrates source modeling with adaptive ANS coding, solving a longstanding problem and enhancing compression performance.

Findings

Achieves high encoding and decoding speeds.

Provides compression ratios close to Shannon entropy.

Enables implementation of adaptive ANS for natural language text.

Abstract

Compression based on asymmetric numeral systems (ANS) combines high encoding and decoding speeds with a compression ratio close to Shannon entropy, while forward modeling of the information source makes it possible to obtain an estimated compressed message size that is less than the entropy. This paper proposes combining these modeling and adaptive coding methods. In addition to ensuring high data processing speeds and compression ratios, this approach enables one to implement the adaptive ANS, which has long remained an important scientific and practical problem.