From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms

TL;DR

This paper benchmarks 17 multi-modal large language models on a challenging handwritten medical form dataset, showing recent models achieve around 85% accuracy and highlighting strengths and improvements in digitising complex handwritten data.

Contribution

It provides the first comprehensive benchmarking of state-of-the-art multi-modal LLMs on real-world handwritten form digitisation, revealing their capabilities and limitations.

Findings

Latest models reach ~85% accuracy on complex handwritten forms.

Prompt optimisation improves macro metrics by over 60%.

GPT 5.4 excels in noisy date extraction and reliability.

Abstract

Manual digitisation of structured handwritten documents is slow and costly. We benchmark 17 leading frontier multi-modal large language models and open-source models against a very challenging real-world medical form that mixes dates; structured, printed text; hand-written responses and significant variability challenges. None of the smaller or older models perform well but the latest Google and OpenAI models reach accuracies around $85\%$ with weighted F1 scores $\simeq 90\%$ across the discrete or predefined fields despite the very challenging nature of the responses. Clear task specific strengths emerge: GPT 5.4 excels in noisy date extraction as well as reliability with the lowest hallucination rate ($6\%$). Claude Sonnet 4.6 had the best average performance across formatted fields (dates and numerical values), while Gemini 3.1 delivered the best overall performance, with the lowest free text error rates (WER = $0.50$ and CER = $0.31$) and the strongest results across discrete classification metrics. We further show that prompt optimisation dramatically improves macro precision, recall and F1 by over $60\%$, but has little impact on weighted metrics (only $\sim2-5\%$ improvement). These results provide evidence that the rapid improvements of multimodal large language models offer a compelling pathway toward fully automated digitisation of complex handwritten workflows that is particularly relevant in low- and middle-income countries.