Measuring Human Perception to Improve Handwritten Document Transcription

TL;DR

This paper introduces a novel loss function incorporating psychophysical measurements of human perception to enhance deep learning models for handwritten document transcription, demonstrating improved accuracy on multiple datasets including historical manuscripts.

Contribution

It proposes a new loss formulation that integrates human perceptual data into training deep neural networks for handwriting recognition, applicable to modern and historical documents.

Findings

Performance improved on IAM and RIMES datasets

Demonstrated feasibility on 9th-century Latin manuscripts

Applicable across different neural network architectures

Abstract

The subtleties of human perception, as measured by vision scientists through the use of psychophysics, are important clues to the internal workings of visual recognition. For instance, measured reaction time can indicate whether a visual stimulus is easy for a subject to recognize, or whether it is hard. In this paper, we consider how to incorporate psychophysical measurements of visual perception into the loss function of a deep neural network being trained for a recognition task, under the assumption that such information can enforce consistency with human behavior. As a case study to assess the viability of this approach, we look at the problem of handwritten document transcription. While good progress has been made towards automatically transcribing modern handwriting, significant challenges remain in transcribing historical documents. Here we describe a general enhancement strategy, underpinned by the new loss formulation, which can be applied to the training regime of any deep learning-based document transcription system. Through experimentation, reliable performance improvement is demonstrated for the standard IAM and RIMES datasets for three different network architectures. Further, we go on to show feasibility for our approach on a new dataset of digitized Latin manuscripts, originally produced by scribes in the Cloister of St. Gall in the the 9th century.

Figures

Tables

Table 1. TABLE I: Examples of off-the-shelf tools that could be used for a handwritten Latin transcription task. Character (CER) and word (WER) error rates above 100 % percent 100 100\% can occur when the predicted strings are longer than the correct solution, causing more insertion errors than the length of the string.

	Performance on Latin Dataset from Sec. IV
	Validation Set		Test Set
	CER	WER	CER	WER
Tesseract [17]	49.65%	105.40%	44.42%	104.80%
Ocropus [18]	53.02%	113.17%	53.02%	113.17%
FineReader [19]	66.11%	98.48%	70.19%	97.25%
Ocular [25]	103.84%	114.23%	96.32%	115.64%

Table 2. TABLE II: A comparison of the average results on the Validation Set of each dataset for each set of experiments. “I” stands for IAM, “R” for RIMES, and “L” for the Latin dataset. “Raw Improvement” refers to the number of characters or words that performance improved by when using the psychophysical loss. “ERR” is standard error. Improvement is observed in all but one case.

	Average Results on Validation Sets
	CRNN
	Without Psychophysical Loss				With Psychophysical Loss (Easy Priority)				% Imp.		Raw Imp.
	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	WER (%)	CE	WE
I	6.38	$\pm$0.11	20.60	$\pm$0.31	6.16	$\pm$0.13	19.96	$\pm$0.32	3.39	3.04	111.4	40.2
R	7.29	$\pm$0.36	18.61	$\pm$0.95	6.13	$\pm$0.16	15.41	$\pm$0.58	15.52	16.89	997.4	526.4
L	8.65	$\pm$0.07	36.69	$\pm$0.18	8.37	$\pm$0.05	36.25	$\pm$0.39	3.17	1.19	17.6	3.8
	URNN
	Without Psychophysical Loss				With Psychophysical Loss (Easy Priority)				% Imp.		Raw Imp.
	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	WER (%)	CE	WE
I	9.20	$\pm$3.59	28.91	$\pm$9.26	8.74	$\pm$3.23	28.04	$\pm$8.56	3.48	2.18	155.3	68.3
R	6.66	$\pm$0.31	18.55	$\pm$1.11	6.53	$\pm$0.45	17.94	$\pm$1.53	2.05	3.58	44.3	43
L	12.37	$\pm$0.55	49.16	$\pm$2.01	11.64	$\pm$0.81	47.88	$\pm$2.95	6.08	2.77	60	16.3
	CTransformer
	Without Psychophysical Loss				With Psychophysical Loss (Easy Priority)				% Imp.		Raw Imp.
	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	WER (%)	CE	WE
I	10.52	$\pm$0.19	34.86	$\pm$0.51	10.27	$\pm$0.12	34.47	$\pm$0.32	2.33	1.07	17.6	3.8
R	7.41	$\pm$0.16	21.06	$\pm$0.55	7.07	$\pm$0.10	18.86	$\pm$0.87	4.53	10.08	187.6	443
L	12.13	$\pm$0.15	54.93	$\pm$0.33	11.88	$\pm$0.12	55.39	$\pm$0.92	2.05	-0.85	10	-7.6

Table 3. TABLE III: A comparison of the average results on the Test Set of each dataset for each set of experiments. All cases improve.

	Average Results on Test Sets
	CRNN
	Without Psychophysical Loss				With Psychophysical Loss (Easy Priority)				% Imp.		Raw Imp.
	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	WER (%)	CE	WE
I	8.00	$\pm$0.10	24.49	$\pm$0.29	7.76	$\pm$0.14	23.70	$\pm$0.37	2.95	3.19	445.6	285.6
R	7.47	$\pm$0.36	19.28	$\pm$0.95	6.33	$\pm$0.15	16.15	$\pm$0.67	16.37	16.04	961.8	558.6
L	6.78	$\pm$0.06	32.11	$\pm$0.29	6.45	$\pm$0.04	30.80	$\pm$0.31	4.77	4.06	40.4	25.2
	URNN
	Without Pyschophysical Loss				With Pyschophysical Loss (Easy Priority)				% Imp.		Raw Imp.
	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	WER (%)	CE	WE
I	11.28	$\pm$4.09	33.03	$\pm$9.60	10.93	$\pm$3.80	32.48	$\pm$8.99	2.25	1.00	93	109
R	6.75	$\pm$0.31	18.86	$\pm$1.00	6.66	$\pm$0.47	18.36	$\pm$1.61	1.56	3.03	485.7	182.7
L	8.98	$\pm$0.29	41.18	$\pm$1.11	8.59	$\pm$0.40	40.07	$\pm$1.74	4.40	2.76	60.3	25.3
	CTransformer
	Without Pyschophysical Loss				With Pyschophysical Loss (Easy Priority)				% Imp.		Raw Imp.
	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	ERR (%)	WER (%)	ERR (%)	CER (%)	WER (%)	CE	WE
I	13.06	$\pm$0.21	39.93	$\pm$0.47	12.79	$\pm$0.11	39.49	$\pm$0.28	1.98	1.05	461	164.4
R	7.58	$\pm$0.18	21.68	$\pm$0.62	7.27	$\pm$0.12	20.62	$\pm$0.48	4.01	4.76	309.8	200
L	9.81	$\pm$0.16	49.59	$\pm$0.68	8.96	$\pm$0.07	47.17	$\pm$0.48	8.57	4.82	99.6	46.8

Table 4. TABLE IV: Average results for the control experiments on the IAM dataset using the CRNN architecture.

	IAM Validation (%)				IAM Test (%)
	CER	ERR	WER	ERR	CER	ERR	WER	ERR
B	6.38	$\pm$.11	20.60	$\pm$.31	8.00	$\pm$.10	24.49	$\pm$.29
E	6.16	$\pm$.13	19.96	$\pm$.32	7.76	$\pm$.14	23.70	$\pm$.37
H	6.00	$\pm$.03	19.62	$\pm$.08	7.56	$\pm$.02	23.41	$\pm$.03
M	6.16	$\pm$.06	20.33	$\pm$.19	7.76	$\pm$.07	24.02	$\pm$.21
C	5.73	$\pm$.10	19.11	$\pm$.27	7.68	$\pm$.11	23.92	$\pm$.29
L	6.28	$\pm$.11	20.33	$\pm$.21	7.92	$\pm$.08	24.21	$\pm$.19

Table 5. TABLE V: Comparison to other work on handwritten document transcription with language model post-correction.

	IAM Val. (%)		IAM Test (%)
	CER	WER	CER	WER
Voigtlander et al. [26]	2.4	7.1	3.5	9.3
Puigcerver et al. [27]	2.9	9.2	4.4	12.2
Xiao et al. [31]	1.96	6.63	3.03	8.66
CRNN w/o Psych. Loss	3.00	8.72	4.71	12.6
CRNN w/ Psych. Loss (E)	2.79	8.19	4.35	11.09
CRNN w/ Psych. Loss (H)	2.88	8.29	4.21	10.68

Equations

$${\mathcal{L}}_{\mathrm{CTC}}=-\sum _{x_i,{\mathbf{l}}_i\in \mathcal{X}}\log p({\mathbf{l}}_i|{\mathbf{y}}_i)$$

$$z_i=\{\begin{array}{ll}m-r_i,& \text{if easy examples prioritized}\\ r_i,& \text{otherwise}\end{array}$$

$${\mathcal{L}}_{\mathrm{psych}}=-\sum _{x_i,{\mathbf{l}}_i,{ϵ}_i,z_i\in \mathcal{X}}\left(\log p({\mathbf{l}}_i|{\mathbf{y}}_i)+(-{ϵ}_i\times z_i)\right)$$

Full text

I Introduction

In archives scattered around the globe, old manuscripts can be found piled up to the ceiling and spread out as far as the eye can see. The amount of writing produced on physical media since antiquity is staggering, and very little of it has been digitized and transcribed into plain text for researchers to study using modern data mining tools [1]. Work in the digital humanities has sought to address this problem by deploying everything from off-the-shelf optical character recognition (OCR) tools [2] to state-of-the-art convolutional neural network-based transcription pipelines [3]. However, such work has been underpinned by the long-standing, yet incorrect, belief that computer vision has solved handwritten document transcription [4, 5]. The open nature of this problem, coupled with a difficult data domain that has largely remained the realm of specialist scholars, makes it a fascinating case study for testing the capabilities of artificial intelligence.

For historical documents, work is ongoing in computer vision [6, 7, 4] and natural language processing (NLP) [8, 5] to produce a method that operates with little human intervention. However, to solve this problem, it helps to take a closer look at the behavior of reading. In this paper we seek out a fresh perspective by addressing three understudied aspects of the problem related to the way humans perceive handwritten documents: (1) the need in some cases to collect annotations for supervised learning from experts, (2) a loss formulation that makes use of behavioral measurements from people, and (3) a recognition pipeline that can successfully operate in a purely visual mode when it is not possible to train a language model on the statistics of language for post-correction.

The conventional wisdom in machine learning says that to improve performance on a task, more data should be collected. However, when it comes to document analysis tasks that require some form of expertise to produce supervised labels, it is not possible to turn to workers on Amazon’s Mechanical Turk service to annotate a dataset. For example, if the task is to transcribe a parchment manuscript written in Latin, one will observe a large difference in behavior between the novice and experienced reader, with the novice producing an unreliable transcription. Researchers within the digital humanities have been struggling with the problem of a lower annotation yield for years [9]. In this work, we explore a new data annotation regime that allows us to collect more information from fewer annotators. The key is in the data we collect: images with more information-rich annotations.

Turning to the study of perception, annotations that are not basic labels, but instead measurements of the internal processes of the human visual system, have been shown to be effective for improving supervised machine learning [10]. The strategy relies on the use of visual psychophysics, a set of methods and procedures from psychology that are used to study the relationships between physical stimuli and mental phenomena. For instance, the amount of time it takes a subject to recognize a handwritten character is correlated with the quality of the handwriting under scrutiny. Including this information into the training regime of a machine learning algorithm gives the resulting model some notion of what is difficult, and what is easy. This is critical for problems that contain images of varying degrees of difficulty. Previous work has looked at conditioning classifiers on psychophysical data [10], but has not considered using such information for representation learning. Here we introduce a loss formulation for deep networks based on perceptual measurements (Fig. 1).

The goal of current work in handwritten document transcription is to produce a useful tool that can generalize to new documents, given appropriate training data. Prior approaches have focused on individual aspects of the problem (e.g., image restoration [11], character segmentation [3], word recognition [12], data augmentation [13], language modeling [14]). However, NLP components that require dictionaries and other comprehensive corpora are problematic in some contexts. For instance, a separate neural language model is often trained to match the source language of the documents being considered. It takes the output of a vision-based transcription network and makes corrections based on a learned probability distribution of character- or word-level sequences. For historical documents, a challenge exists in training language models under circumstances where incomplete knowledge of the lexicon is always present (e.g., a complete dictionary of Medieval Latin has never been compiled). In spite of this, we show that it is still possible to improve the performance of an imperfect vision system by making use of psychophysical measurements.

Contributions. To summarize, the contributions of this paper are: (1) An open source software platform for collecting crowdsourced psychophysical and conventional annotations for historical documents, designed with the humanities scholar in mind. (2) A novel loss formulation incorporating measurements of human vision into the training process of three different deep neural networks. (3) A new collection of data for transcription evaluation including psychophysical annotations for partitions of the IAM [15] and RIMES [16] datasets, as well as a new dataset of open access Medieval Latin manuscript images and annotations. (4) Experiments showing that the proposed loss formulation reliably improves the performance of different network architectures on IAM, RIMES, and the new Latin manuscript set.

II Prior Work

Recent Advances in Document Transcription. For general use, off-the-shelf OCR packages like Tesseract [17], OCRopus [18], and ABBYY FineReader PDF [19] remain popular. These packages are often the first choice of scholars working with historical documents because they are easy to use [20]. However, because they have been designed for use with modern print documents, their performance is often very poor on documents outside of that domain. This can be seen in Table I, which contains results for a transcription task that used the Latin manuscript dataset described in Sec. IV to evaluate these packages. Tesseract in this case was trained with Latin data [21], and FineReader configured to process Latin. Even with these capabilities, the performance is unacceptably low.

The most important recent advances in document transcription are based on deep learning or probabilistic generative models. Poznanski and Wolf [12] introduced a feed-forward Convolutional Neural Network (CNN) that estimates the lexical attributes (i.e., n-gram features) of a word for transcription purposes. Doetsch et al. [22] and Bluche et al. [23] explored recurrent approaches for sequence processing by using Bidirectional Long Short-Term Memory (LSTM) in different input and architectural configurations. Bukhari et al. [24] addressed the problem of labeled data scarcity in historical document datasets by using image segmentation and character clustering to automatically generate data for LSTM training. With respect to generative models, Berg-Kirkpatrick proposed jointly modeling the text of the document and the noisy process of rendering glyphs in an unsupervised manner [8]. The implementation is available as an open source software package called Ocular [25], but is better suited to print documents (see Table I).

When it comes to the state-of-the-art on standard benchmark datasets for document transcription like IAM and RIMES, neural networks are combined with domain-specific heuristics for the target language and writing style. For instance, Multidimensional LSTM (MDLSTM) networks have been paired with $n$-gram language models trained over large-scale corpora containing a comprehensive lexicon [26, 27]. Some of the very best results on historical documents [28, 13] have been achieved using the Convolutional Recurrent Neural Network (CRNN) architecture of Shi et al. [29] and a Connectionist Temporal Classification (CTC) loss [30] for training as a basis. The CRNN is able to support image-based sequence recognition in an end-to-end trainable fashion. Xiao et al. [31] proposed a number of improvements to CRNN including rectification that corrects the character distortions that are prevalent in the IAM dataset, as well as post-processing by a word beam search language model. We will show that training with our proposed loss function can improve CRNN performance without comprehensive knowledge of the target language or changes to the architecture.

Computer Vision for Manuscript Studies. Images of Medieval manuscripts provide a unique challenge for computer vision algorithms, given the prevalence of poor image quality, inconsistent handwriting, and often incomplete knowledge about the language at hand. Work on this problem is highly interdisciplinary, where the end goal is to provide a useful tool to scholars in the humanities. Thus engineering has gone into creating end-to-end processing pipelines that are accessible to paleographers [6, 3, 32, 33, 34]. Early work in computer vision for manuscript studies articulated the problem as one of sequence processing over lines from the images, where approaches like HMMs apply if trained on the target language [35, 36]. Secondary problems in data quality were noted, and various approaches to fix them have been proposed [37, 11, 38].

As in automated transcription for modern texts, neural network-based approaches are now routinely deployed for manuscript analysis. Fischer et al. [39] demonstrated the effectiveness of a Bidirectional LSTM in transcribing a 13th century manuscript written in Middle High German. Firmani et al. [3] showed over a corpus of manuscripts from the Vatican Apostolic Archive that if character segmentation is accurate, a CNN can be used to classify characters with high accuracy. Related work was undertaken to classify Medieval handwriting for archival cataloguing [7]. But error rates remain stubbornly high, thus leaving humans in the loop to make corrections.

Visual Psychophysics for Computer Vision. Work in this direction has looked at the sensitivity of different visual systems to the local statistics of natural images [40], the behavioral consistency of deep networks to human vision [41, 42], and human vs. non-human primate vs. deep network object recognition performance [43, 44]. A psychophysics-inspired evaluation framework has been proposed for object recognition in computer vision [45]. Most closely related to the work in this paper is that of Scheirer et al. [10], where a loss function that applies penalties based on psychophysical annotations attached to individual examples during classifier training is used to improve classification accuracy. In contrast to classification methods that gradually emphasize more difficult examples like boosting [46], a psychophysical loss function exploits precise measurements of human vision that are common in psychology, but not in computer vision.

There are four key distinctions between our work and that of Scheirer et al. [10]. (1) We extend their strategy, which was limited to Support Vector Machine (SVM) classifiers, to representation learning by introducing a loss formulation for deep networks that makes use of psychophysical measurements. (2) The proposed loss function is more sophisticated than the loss function introduced by Scheirer et al., operating in two modes depending on what type of examples (easy or hard) should be prioritized during training. (3) This work looks at the collection of psychophysical measurements from experts (in this case readers of Latin), as opposed to non-experts on a crowdsourcing platform. (4) This work concentrates on the specific application of handwritten document transcription, while prior work has limited itself to face detection.

III A Loss Formulation That Makes use of Measurements from Human Readers

The approach for handwritten document transcription we consider assumes an end-to-end pipeline composed of a set of distinct modules for computer vision and natural language processing. We begin with a data collection process incorporating expert readers making use of a custom psychophysical data collection tool. This provides data that can be used by a standard handwritten text recognition model, but also allows us to collect additional data that measures the process of reading. These psychophysical measurements of human vision can be incorporated into a loss function that improves performance. Here we describe the new loss formulation, explaining how to go from psychophysics experiments to a training strategy for any CTC-based transcription pipeline111All code and data will be released after publication..

Psychophysical Data Collection. Historical documents present unique challenges for handwritten document transcription algorithms. For instance, while there are thousands of high-quality images of Medieval manuscripts available on the Internet, very few of those documents have been annotated for transcription tasks. Even more problematic, such documents cannot be annotated by just anyone — paleographic experience is often required. In a pilot study, we had a reader who was unfamiliar with the Latin language transcribe a manuscript page from the 9th century. This reader’s result was scored against the transcription produced by a knowledgeable reader of Latin, and had a character error rate of $35\%$. Therefore, this task cannot simply be farmed out to Amazon’s Mechanical Turk service. It requires significant subject matter knowledge. Furthermore, even when an expert does take the time to transcribe manuscript images, they often make editorial changes to expand abbreviations, and correct spelling errors and other abnormalities in the text, making the resulting ground-truth transcription unsuitable for training a system that can produce an exact (i.e., diplomatic) transcription.

To address these challenges we developed a web-based tool to leverage expert reading behavior that not only produces exact ground-truth data, but also collects psychophysical measurements of the process of reading. The tool is based on a version of the Image Citation Tool from the Homer Multitext Project [47], which provides a familiar interface to non-technical annotators working on digital data curation for historical documents. An additional stage was added to collect psychophysical annotations. In terms of the workflow, an expert reader is shown a screen in which they are prompted to identify a single character that has been hand-segmented from the page. The reader is then timed as they select which character is represented in the image from a pre-defined set of characters that cover all of the possibilities for a dataset. Each character can be shown to multiple readers, with their reaction times averaged to account for environmental differences.

This type of psychophysical data collection is not just constrained to expert reading problems. Psychophysical annotations for modern languages such as English or French can be collected in a manner similar to the above process, but with a bit more flexibility. Since most people who are fluent in a Western language can easily read print and cursive handwriting, there is no need to present them with segmented characters. For modern languages, the reader types out a candidate line as they parse it. While each line is being transcribed, the individual keystrokes were timed and recorded. The data are collected in small batches of 25 images, with 5 control images that the rest of the measurements are normalized against to control for environmental factors across readers, such as different typing speeds. Supp. Mat. Sec. 1 contains additional details and screenshots of the platform.

Psychophysical Loss Function. The tools described above can measure timing information at the character- or line-level. In this work, we choose to operate at the line-level with respect to input to the transcription system. Character-level measurements are averaged across a line to normalize for length. Thus each line image has an associated average reaction time $r$ (see Supp. Table 1). This form of psychophysical measurement is inspired by the work of Scheirer et al. [10], which demonstrated that utilizing reaction time as a psychophysical measurement can improve the performance of SVM. Other psychophysical measurements could be used as well, including accuracy (if ground-truth is available before the behavioral measurements are taken) and eye movements.

In this work, we propose a loss formulation for training CNNs to perform handwritten document transcription. In order to facilitate this, we collect psychophysical measurements from a subset of lines from each dataset and use them to derive penalties that will be applied by the loss function during training. In practice, these measurements can help the network prioritize the importance of examples that are easier or harder for humans to transcribe, depending on the application.

In order to make use of the psychophysical data that was collected, we designed a new loss function that is a variation of the commonly used CTC loss function originally introduced by Graves et al. [30]. This is the standard loss function for transcription tasks. CTC loss is the negative logarithm of the probability of the ground-truth given a predicted text sequence. After calculating this loss for a specific training image, our method adds a psychophysical penalty that is scaled to the character error rate of the output.

Assume we have a training data set ${\cal X}=\{x_{i},\mathbf{l}_{i},\epsilon_{i},z_{i}\}$, where $x_{i}$ is the training image, $\mathbf{l}_{i}$ is the associated ground-truth label sequence, $\epsilon_{i}$ is the associated character error rate, and $z_{i}$ is an associated psychophysical penalty. The objective when using CTC loss is to minimize the negative log-likelihood of the conditional probability of the ground-truth:

[TABLE]

where $\mathbf{y}_{i}$ is the sequence produced by the recurrent and convolutional layers of a network when processing $x_{i}$.

To reformulate the above loss to incorporate the psychophysical measurements, we must first calculate a psychophysical penalty for each image that matches our assumptions about visual perception and reading (Fig. 2). Assume that we have a set $R=\{r_{1},\ldots,r_{n}\}$ of sorted average reaction time measurements from an experiment. The maximum value of that set is $m=\mathrm{max}(R)$. Also assume that images with a short measured reaction time (in ms.) are easy for a human observer to recognize characters in, and images with a long measured reaction time are more difficult. To calculate a psychophysical penalty $z_{i}$ that prioritizes easy examples during training, an average reaction time $r_{i}$ is subtracted from $m$. To prioritize hard examples (as in hard example mining [48, 49]), $z_{i}$ is simply assigned to $r_{i}$.

[TABLE]

A loss function conditioned on the psychophysical measurements should have the effect of pushing the training in a direction that will perform better on the prioritized examples, with priority specified by a parameter $\rho$. It should also preserve the penalties for incorrect predicted text sequences. Thus we combine the CTC loss (as defined in Eq. 1) with an additional loss term derived from the psychophysical measurements and the character error rate associated with an image:

[TABLE]

IV Experimental Setup

Datasets. As a baseline for assessing the effect of training with the proposed loss function, we consider two standard databases: IAM [15] for English and RIMES [16] for French. The IAM database contains more than $10,000$ images of handwritten lines of text created by $657$ writers. The texts those writers transcribed are from the Lancaster-Oslo/Bergen Corpus of British English [50]. The dataset has been split according what is known as the Aachen split, specifically the Aachen split as defined by Puigcerver et al. in [51]. It has a training set ($6,161$ lines), a validation set ($966$ lines), and a test set ($2,915$ lines). The RIMES database consists of images of lines of text from letters composed by volunteers. While the dataset does not have an official training and validation split, we used the same split described by Puigcerver et al. [51], which is standard practice in the literature. This split has a training set ($1,0171$ line), validation set ($1,161$ images), and testing set ($778$ images). For the challenging historical documents that we are primarily concerned with, we created a new set of open access Latin language documents. Our Latin dataset comes from e-codices, which is the Virtual Manuscript Library of Switzerland [52]. The specific codex selected is from the Cloister of St. Gall [53]. We collected $1,861$ images of handwritten lines that were split into a training set ($1,479$ images), a validation set ($196$ images), and a test set ($186$ images) in a page disjoint fashion. It is not known how many writers this set consists of, but in practice, all scribes wrote in a deliberately similar script.

We collected psychophysical annotations for $2,132$ text lines from IAM, which is approximately $35\%$ of the training set. For RIMES, we collected psychophysical annotations for $1,257$ lines, which is approximately $12\%$ of the training set. These numbers reflect the amount of data that we were able to collect through psychophysics experiments during the time period this research was conducted. The psychophysical annotations for IAM were collected from graduate students fluent in English. The RIMES annotations were collected from graduate students with a working knowledge of French. With respect to the Latin dataset, the texts were annotated by graduate students who were fluent in Latin and trained in paleography. We asked our annotators to create exact transcriptions from the lines they were provided. This dataset has psychophysical annotations across the entire training set of $1,479$ images.

Handwritten Text Recognition Networks. In order to demonstrate improvement across different neural network architectures, we ran our experiments with three different handwritten text recognition networks. The first is the standard CRNN architecture. The implementation we used was identical to the one used by Puigcerver in [27]. It consists of five convolutional layers and three bidirectional LSTM layers. The second architecture was a variation on the CRNN, where the convolutional layers were replaced with a U-Net [54] segmentation module. Because of the added complexity of adding the U-Net, the number of LSTM layers was reduced to two. We refer to this architecture as URNN. The third architecture was also a variant of CRNN, where the recurrent layers were replaced with a transformer encoder [55] with eight heads. This model is referred to as CTransformer. Additional details for these architectures can be found in Supp. Mat. Sec. 2.

For the CRNN and CTransformer networks, we generated five random initializations and trained each network with both the standard CTC loss function and our proposed psychophysical loss. The URNN model takes an order of magnitude longer to train, so we followed the same procedure, but with only three random initalizations. Each experiment uses the same training configuration, stopping after $80$ epochs without improvement. For the CRNN and CTransformer, we cut the learning rate in half every $15$ epochs without improvement, but we found that this made the URNN fail to converge, so we increased this interval to $30$ epochs for that network. For each architecture we used the optimizer found to be most effective, which was RMSProp for CRNN, Adadelta for URNN, and Adam for CTransformer. A difference from other setups found in the literature for the experiments performed on IAM and RIMES is that we reduced the image height to $64$ from the typically used $128$ to increase processing speed. This had a slight negative impact on overall performance, but did not affect the assessment of the loss function effect.

V Experimental Results

Measurable Effect of the Psychophysical Loss. The primary set of experiments assessed the change in network performance after the move to the new loss function. An overview of the results with priority on easy examples can be seen in Tables II and III. These tables show the average results for each set of experiments by model and dataset. A complete set of the $88$ individual experiments that were run is included in Supp. Mat. Sec. 3. We also include tables in the Supp. Mat. that show the best result from each experiment. With respect to the overall measurable effect, we find that the psychophysical loss function with priority on easy examples gives a consistent (across datasets and architectures) and repeatable (across trials within a dataset for an architecture) improvement to handwritten text recognition performance.

Across the $44$ pairs of experiments we performed on identical initializations with and without the psyschophysical loss function, we found that the models trained using the new loss preformed better (in terms of Test Set CER) set in $40$ of them. Every error rate is improved on average with the psychophysical loss, except for the WER of the CTransformer on the Latin validation set. It is also worth noting that the best overall result on the test sets from each set of experiments was always from a model trained using the psychophysical loss function (see Supp. Tables 2, 3 and 4).

Experiments were also performed putting the priority on hard examples for the psychophysical loss function (Supp. Tables 7 & 8). The results for this mode of the proposed loss function are mixed compared to putting the priority on easy examples. In some cases lower error rates are achieved (e.g., CRNN on all datasets), but in other cases the error rates are higher than the baseline (e.g., CTransformer on all datasets). One possible explanation for the consistent improvement observed for easy priority may be the normalizing effect induced by subtracting every measurement from the maximum average reaction time. Another possible explanation is that the hard examples are not as reliably correctable as the easy examples during training. The loss function priority should be treated as a hyperparameter, with a decision on the priority made after evaluating both modes using a validation set.

There are two key differences in these experiments compared to other recent work in document transcription. The results shown in Tables II and III do not make use of a language model as part of the transcription pipeline. The primary reason for excluding a language model is that due to our specific interest in historical documents with large amounts of novel language content, it is not possible to train an effective one. Further, these experiments do not make use of any pre-training on synthetic data, since it is not possible to gather psychophysical annotations for synthetic handwritten text images generated on the fly. Nevertheless, the raw improvement counts are substantial, and represent a marked improvement for the reader of the transcriptions. Qualitative examples can be found in Supp. Mat. Sec. 5.

Given the frequent need in the digital humanities to transcribe very large archives of documents, we also made some decisions that optimize for speed rather than overall performance. This helped in processing the large number of folds run for these experiments. As mentioned earlier, we used an image height of $64$ pixels rather than the more standard $128$. This had a notable impact on reducing computation time. Using a smaller network size to process smaller input images cuts the computation time from approximately $600$ seconds per epoch to about $250$. At inference time, the $64$ pixel input height allows for the processing of $31$ images per second, while the $128$ pixel input height yields $16$ images per second. These numbers were calculated using one NVIDIA Titan Xp GPU.

Control Experiments. We performed a series of control experiments to further verify that psychophysical measurements carry a useful signal for machine learning (with and without the proposed loss function), and to investigate if the proposed loss function is useful for penalties derived from data other than psychophysical measurements. All of the following experiments were run from the same five random initializations of the CRNN architecture from the set of experiments described above for the IAM dataset. Full details for the setup of each of these experiments can be found in Supp. Mat. Sec. 6. All results are in Table IV, with the baseline results in the row labeled ‘B’, and the results using the psychophysical loss with psychophysical measurements labeled ‘E’ for priority on easy examples and ‘H’ for priority on hard examples.

The first control experiment examined the use of algorithmic hard example mining instead of psychophysical measurements using the proposed loss function. Hard examples have proved to be useful in other vision applications (e.g., [48, 49]), and their compatibility with the proposed loss formulation makes them an interesting test case. To do this, we took the baseline CRNN results from each fold, averaged the character error rates for each image, and used those averages as the associated penalties for the images. The results (the row labeled ‘M’) show that by using hard examples, improvement over the baseline CRNN (‘B’) is possible. However, next to the new loss function with psychophysical measurements (‘E’ & ‘H’), the results are comparable (i.e., within the error of ‘E’ and close to ‘H’) for CER, and a bit worse for WER. Thus the new loss works for data other than psychophysical measurements.

The second control experiment assessed the utility of psychophysical measurements in a curriculum learning regime [56], which also considers example difficulty. A curriculum was defined by taking all of the available psychophysical measurements for the IAM dataset, sorting them by reaction time per character, and then dividing them into five groups based on this ordering of difficulty. Groups were incrementally introduced in training after performance on the validation set did not improve for 20 epochs. The results (the row labeled ‘C’) show improvement over the baseline CRNN results (‘B’). This demonstrates that the psychophysical measurements can help machine learning training without the need of a specialized loss function. The results are also comparable to the psychophysical loss function (‘E’ & ‘H’), which is promising for future work combining psychophysics with machine learning.

The third control experiment assessed another measure of example difficulty. We replaced the psychophysical measurements for IAM with the text length of each example (shorter assumed to be easier). We then trained using the psychophysical loss as we did for the experiments reported for the psychophysical data. The results (the row labeled ‘L’) show some improvement over the baseline CRNN (‘B’) — but not significant improvement. The results from this measure were not as good as those using psychophysical measurements (‘E’ & ‘H’), or the hard example mining approach (‘M’).

Larger Input, Language Model and Synthetic Pre-Training. While the previously described networks trade-off speed for accuracy by using a small image height, their performance gets closer to the current state-of-the art results on modern language datasets when using a larger image height (128 pixels) and adding language model post-correction and synthetic pre-training. Language models are an important strategy for achieving good performance on modern language datasets like IAM, but do not work well for languages where we lack a comprehensive lexicon — like Medieval Latin. This is not our primary use case, but is included for comparison.

The language model we used is a word beam search [57]. Synthetic data for the transcription network was generated using the same methodology as Xiao et al. [31]. Training details for the language model and transcription network can be found in Supp. Mat. Sec. 7. As can be seen in Table V, our approach (hard priority ‘H’) yields the third best overall results on the IAM validation and test sets. This is achieved with a far simpler architecture (the original CRNN) than the approach proposed by Xiao et al. The proposed loss reduced error as expected (with easy ‘E’ and hard ‘H’ priority).

In Table V, comparison results are reported as they appear in published papers. This means that things like the language models used for post-correction and pre-possessing steps differ between approaches. Also, with the exception of the experiments reported by Puigcerver et al. [27], error bars are not reported, making it difficult to tell what the effect of changing the seed for initialization is (a major motivation for the use of multiple folds in our experiments above). For instance, our CRNN model without using the psychophysical loss is identical to the “baseline” and “Recogm” models used in the ablation studies by Puigcerver et al. and Xiao et al. [31] — but all three implementations differ somewhat in performance.

VI Discussion

The motivation for this work has been the general need for a useful handwritten document transcription tool for the user who is not familiar with computer vision, but may be an expert in another field looking at challenging or unusual images. What did we do differently than previous approaches to get closer to fulfilling this need? In essence, we studied the reading behavior of experts using the methods and procedures of visual psychophysics from the field of psychology. By measuring the time it took experts to recognize particular characters, we gained a better sense of which ones are easy to recognize and which are more difficult. Considering historical documents, small, yet consequential, mistakes can completely alter the meaning of important lines, possibly leading to misinterpretations of literature and history. Avoiding these mistakes is essential for making transcription tools more useful.

To operationalize the above idea, we avoided creating a transcription pipeline for a specific language or style of writing, and instead opted to formulate a general loss strategy for CNNs. We demonstrated that collected psychophysical measurements could be incorporated into the training regime of a character recognition network to improve the accuracy of the transcriptions. With respect to the evaluation of neural network-based handwritten document transcription pipelines, we suggested a strategy that makes use of multiple folds with different random seeds as a form of sensitivity analysis, showing before and after results for changes to the loss function across different languages and many different writers. We also recommended decoupling the vision and language components of transcription pipelines, because for many historical documents the lack of complete dictionaries for the target language leads to less effective language models.

Even with these advances, there is still a ways to go in the development of an off-the-shelf tool like Tesseract for this problem. Subtle shifts in orthographic conventions within a language create ambiguity for current segmentation models. An alternative approach is to make pixel-level annotations available at training time, similar to segmentation in medical imaging applications. This might provide more information from which to disambiguate characters. With respect to language model improvements, there is a need to handle the abbreviations that are common for certain historical periods. In Latin manuscripts, paleographers expand these to the original word when producing transcriptions. For example, in Fig. 1, the abbreviation dnt. should be expanded to the word dicunt. A language model that can do this automatically is needed. Quite to the contrary of common belief, there is still significant and challenging work to be done on handwritten documents.

Acknowledgment

This work was supported in part by the following organizations within the University of Notre Dame: Notre Dame Research, the College of Arts and Letters, the Medieval Institute, the Office of Mission Engagement, and the Office of Digital Learning. Additional support was provided by Adobe Systems and the NVIDIA Corporation. We thank Cana Short, Mihow McKenny, Melody Wauke, Curtis Wigington and Christopher Tensmeyer for their help preparing data and invaluable advice.