CER-HV: A Human-in-the-Loop Framework for Cleaning Datasets Applied to Arabic-Script HTR

TL;DR

This paper introduces CER-HV, a human-in-the-loop framework utilizing CER-based ranking to detect and clean label errors in Arabic-script handwritten text datasets, significantly improving data quality and model performance.

Contribution

The paper presents CER-HV, a novel framework combining CER-based noise detection with human verification, revealing dataset issues and enhancing HTR accuracy across multiple Arabic-script languages.

Findings

Datasets contain significant transcription, segmentation, and orientation errors.

CER-HV improves evaluation CER by 0.3-1.8% depending on dataset noisiness.

State-of-the-art CRNN performance achieved on multiple Arabic-script HTR datasets.

Abstract

Handwritten text recognition (HTR) for Arabic-script languages still lags behind Latin-script HTR, despite recent advances in model architectures, datasets, and benchmarks. We show that data quality is a significant limiting factor in many published datasets and propose CER-HV (CER-based Ranking with Human Verification) as a framework to detect and clean label errors. CER-HV combines a CER-based noise detector, built on a carefully configured Convolutional Recurrent Neural Network (CRNN) with early stopping to avoid overfitting noisy samples, and a human-in-the-loop (HITL) step that verifies high-ranking samples. The framework reveals that several existing datasets contain previously underreported problems, including transcription, segmentation, orientation, and non-text content errors. These have been identified with up to 90 percent precision in the Muharaf and 80-86 percent in the PHTI datasets. We also show that our CRNN achieves state-of-the-art performance across five of the six evaluated datasets, reaching 8.45 percent Character Error Rate (CER) on KHATT (Arabic), 8.26 percent on PHTI (Pashto), 10.66 percent on Ajami, and 10.11 percent on Muharaf (Arabic), all without any data cleaning. We establish a new baseline of 11.3 percent CER on the PHTD (Persian) dataset. Applying CER-HV improves the evaluation CER by 0.3-0.6 percent on the cleaner datasets and 1.0-1.8 percent on the noisier ones. Although our experiments focus on documents written in an Arabic-script language, including Arabic, Persian, Urdu, Ajami, and Pashto, the framework is general and can be applied to other text recognition datasets.