A Transformer Based Handwriting Recognition System Jointly Using Online and Offline Features

TL;DR

This paper introduces a novel end-to-end transformer-based handwriting recognition system that fuses online stroke data and offline images early in the process, leading to improved accuracy and writer independence.

Contribution

The work presents a new joint online-offline handwriting recognition model using early fusion in a shared latent space with transformers, achieving state-of-the-art results.

Findings

Achieves up to 1% accuracy improvement over previous bests.

Demonstrates strong writer independence in recognition.

Successfully adapts to gesturification on the ISI-Air dataset.

Abstract

We posit that handwriting recognition benefits from complementary cues carried by the rasterized complex glyph and the pen's trajectory, yet most systems exploit only one modality. We introduce an end-to-end network that performs early fusion of offline images and online stroke data within a shared latent space. A patch encoder converts the grayscale crop into fixed-length visual tokens, while a lightweight transformer embeds the $(x, y, \text{pen})$ sequence. Learnable latent queries attend jointly to both token streams, yielding context-enhanced stroke embeddings that are pooled and decoded under a cross-entropy loss objective. Because integration occurs before any high-level classification, temporal cues reinforce each other during representation learning, producing stronger writer independence. Comprehensive experiments on IAMOn-DB and VNOn-DB demonstrate that our approach achieves state-of-the-art accuracy, exceeding previous bests by up to 1\%. Our study also shows adaptation of this pipeline with gesturification on the ISI-Air dataset. Our code can be found here.