Transcoding Unicode Characters with AVX-512 Instructions

TL;DR

This paper presents novel algorithms leveraging AVX-512 instructions to efficiently transcode Unicode strings between UTF-8 and UTF-16, achieving significant speed improvements over previous solutions.

Contribution

The authors introduce new algorithms that utilize AVX-512 instructions for Unicode transcoding, providing faster performance and open-source implementation integrated into Node.js.

Findings

Transcode Chinese text at over 5 GiB/s

Achieve twice the speed of previous solutions

Use fewer than 2 CPU instructions per character

Abstract

Intel includes in its recent processors a powerful set of instructions capable of processing 512-bit registers with a single instruction (AVX-512). Some of these instructions have no equivalent in earlier instruction sets. We leverage these instructions to efficiently transcode strings between the most common formats: UTF-8 and UTF-16. With our novel algorithms, we are often twice as fast as the previous best solutions. For example, we transcode Chinese text from UTF-8 to UTF-16 at more than 5 GiB/s using fewer than 2 CPU instructions per character. To ensure reproducibility, we make our software freely available as an open source library. Our library is part of the popular Node.js JavaScript runtime.