Vartani Spellcheck -- Automatic Context-Sensitive Spelling Correction of OCR-generated Hindi Text Using BERT and Levenshtein Distance

TL;DR

Vartani Spellcheck introduces a context-sensitive Hindi spelling correction method combining BERT and Levenshtein distance, significantly improving OCR text accuracy for inflectional Indic languages.

Contribution

The paper presents a novel context-aware Hindi spelling correction approach using BERT and Levenshtein distance, outperforming previous context-free models.

Findings

Achieved 81% correction accuracy on OCR-generated Hindi text.

Significant improvement over previous context-sensitive correction methods.

Demonstrated potential for real-time autocorrect in text editors.

Abstract

Traditional Optical Character Recognition (OCR) systems that generate text of highly inflectional Indic languages like Hindi tend to suffer from poor accuracy due to a wide alphabet set, compound characters and difficulty in segmenting characters in a word. Automatic spelling error detection and context-sensitive error correction can be used to improve accuracy by post-processing the text generated by these OCR systems. A majority of previously developed language models for error correction of Hindi spelling have been context-free. In this paper, we present Vartani Spellcheck - a context-sensitive approach for spelling correction of Hindi text using a state-of-the-art transformer - BERT in conjunction with the Levenshtein distance algorithm, popularly known as Edit Distance. We use a lookup dictionary and context-based named entity recognition (NER) for detection of possible spelling errors in the text. Our proposed technique has been tested on a large corpus of text generated by the widely used Tesseract OCR on the Hindi epic Ramayana. With an accuracy of 81%, the results show a significant improvement over some of the previously established context-sensitive error correction mechanisms for Hindi. We also explain how Vartani Spellcheck may be used for on-the-fly autocorrect suggestion during continuous typing in a text editor environment.