DRetHTR: Linear-Time Decoder-Only Retentive Network for Handwritten Text Recognition

TL;DR

The paper introduces DRetHTR, a decoder-only Retentive Network for handwritten text recognition that achieves faster inference and lower memory usage than Transformers, while maintaining high accuracy by innovative retention mechanisms and layer-wise scaling.

Contribution

It proposes a novel decoder-only Retentive Network architecture with softmax-free retention and layer-wise gamma scaling, enabling linear-time decoding and improved efficiency in handwritten text recognition.

Findings

Achieves 1.6-1.9x faster inference than Transformer baselines.

Reduces memory usage by 38-42% without accuracy loss.

Attains state-of-the-art character error rates on multiple datasets.

Abstract

State-of-the-art handwritten text recognition (HTR) systems commonly use Transformers, whose growing key-value (KV) cache makes decoding slow and memory-intensive. We introduce DRetHTR, a decoder-only model built on Retentive Networks (RetNet). Compared to an equally sized decoder-only Transformer baseline, DRetHTR delivers 1.6-1.9x faster inference with 38-42% less memory usage, without loss of accuracy. By replacing softmax attention with softmax-free retention and injecting multi-scale sequential priors, DRetHTR avoids a growing KV cache: decoding is linear in output length in both time and memory. To recover the local-to-global inductive bias of attention, we propose layer-wise gamma scaling, which progressively enlarges the effective retention horizon in deeper layers. This encourages early layers to model short-range dependencies and later layers to capture broader context, mitigating the flexibility gap introduced by removing softmax. Consequently, DRetHTR achieves best reported test character error rates of 2.26% (IAM-A, en), 1.81% (RIMES, fr), and 3.46% (Bentham, en), and is competitive on READ-2016 (de) with 4.21%. This demonstrates that decoder-only RetNet enables Transformer-level HTR accuracy with substantially improved decoding speed and memory efficiency.