Cleaning Dirty Books: Post-OCR Processing for Previously Scanned Texts

TL;DR

This paper introduces methods for cleaning and deduplicating digitized texts with OCR errors, leveraging language models to improve error detection and correction, and identifying canonical versions of scanned books.

Contribution

It presents novel OCR error correction techniques using language models and alignment data, and demonstrates their effectiveness on large digitized book collections.

Findings

Improved OCR error detection and correction without image data

Identified canonical versions for many scanned books

Corrected over six times more errors than introduced on average

Abstract

Substantial amounts of work are required to clean large collections of digitized books for NLP analysis, both because of the presence of errors in the scanned text and the presence of duplicate volumes in the corpora. In this paper, we consider the issue of deduplication in the presence of optical character recognition (OCR) errors. We present methods to handle these errors, evaluated on a collection of 19,347 texts from the Project Gutenberg dataset and 96,635 texts from the HathiTrust Library. We demonstrate that improvements in language models now enable the detection and correction of OCR errors without consideration of the scanning image itself. The inconsistencies found by aligning pairs of scans of the same underlying work provides training data to build models for detecting and correcting errors. We identify the canonical version for each of 17,136 repeatedly-scanned books from 58,808 scans. Finally, we investigate methods to detect and correct errors in single-copy texts. We show that on average, our method corrects over six times as many errors as it introduces. We also provide interesting analysis on the relation between scanning quality and other factors such as location and publication year.