Patching Leaks in the Charformer for Efficient Character-Level Generation

TL;DR

This paper addresses the information leak problem in Charformer, enabling efficient character-level generation in Transformers, and demonstrates that it can improve training speed and maintain translation quality for morphologically-rich languages.

Contribution

It introduces a solution to prevent information leak in Charformer, allowing effective character grouping in Transformer decoders, and shows benefits in training speed and translation for morphologically-rich languages.

Findings

Charformer downsampling speeds up training by ~30%

No significant translation quality difference with previous methods

Potential benefits for morphologically-rich language translation

Abstract

Character-based representations have important advantages over subword-based ones for morphologically rich languages. They come with increased robustness to noisy input and do not need a separate tokenization step. However, they also have a crucial disadvantage: they notably increase the length of text sequences. The GBST method from Charformer groups (aka downsamples) characters to solve this, but allows information to leak when applied to a Transformer decoder. We solve this information leak issue, thereby enabling character grouping in the decoder. We show that Charformer downsampling has no apparent benefits in NMT over previous downsampling methods in terms of translation quality, however it can be trained roughly 30% faster. Promising performance on English--Turkish translation indicate the potential of character-level models for morphologically-rich languages.