Context-sensitive Spelling Correction Using Google Web 1T 5-Gram Information

TL;DR

This paper introduces a context-sensitive spelling correction method leveraging Google Web 1T 5-gram data, significantly improving error detection and correction accuracy for various types of misspellings in diverse text domains.

Contribution

It presents a novel approach combining error detection, candidate generation, and contextual correction using large-scale web-derived n-gram data, addressing limitations of traditional dictionary-based spell checkers.

Findings

High error correction rate demonstrated across multiple domains.

Significant reduction in non-word and real-word errors.

Potential for parallelization to improve computational efficiency.

Abstract

In computing, spell checking is the process of detecting and sometimes providing spelling suggestions for incorrectly spelled words in a text. Basically, a spell checker is a computer program that uses a dictionary of words to perform spell checking. The bigger the dictionary is, the higher is the error detection rate. The fact that spell checkers are based on regular dictionaries, they suffer from data sparseness problem as they cannot capture large vocabulary of words including proper names, domain-specific terms, technical jargons, special acronyms, and terminologies. As a result, they exhibit low error detection rate and often fail to catch major errors in the text. This paper proposes a new context-sensitive spelling correction method for detecting and correcting non-word and real-word errors in digital text documents. The approach hinges around data statistics from Google Web 1T 5-gram data set which consists of a big volume of n-gram word sequences, extracted from the World Wide Web. Fundamentally, the proposed method comprises an error detector that detects misspellings, a candidate spellings generator based on a character 2-gram model that generates correction suggestions, and an error corrector that performs contextual error correction. Experiments conducted on a set of text documents from different domains and containing misspellings, showed an outstanding spelling error correction rate and a drastic reduction of both non-word and real-word errors. In a further study, the proposed algorithm is to be parallelized so as to lower the computational cost of the error detection and correction processes.