Neural Ctrl-F: Segmentation-free Query-by-String Word Spotting in Handwritten Manuscript Collections

TL;DR

This paper introduces Ctrl-F-Net, a deep neural network model for segmentation-free query-by-string word spotting in handwritten manuscripts, significantly improving search accuracy in historical documents.

Contribution

The paper presents an end-to-end trainable neural network model that outperforms existing segmentation-free methods for handwritten word spotting.

Findings

Outperforms previous state-of-the-art methods on benchmark datasets

Effective in challenging historical handwritten texts

Useful for real-world applications like historical research

Abstract

In this paper, we approach the problem of segmentation-free query-by-string word spotting for handwritten documents. In other words, we use methods inspired from computer vision and machine learning to search for words in large collections of digitized manuscripts. In particular, we are interested in historical handwritten texts, which are often far more challenging than modern printed documents. This task is important, as it provides people with a way to quickly find what they are looking for in large collections that are tedious and difficult to read manually. To this end, we introduce an end-to-end trainable model based on deep neural networks that we call Ctrl-F-Net. Given a full manuscript page, the model simultaneously generates region proposals, and embeds these into a distributed word embedding space, where searches are performed. We evaluate the model on common benchmarks for handwritten word spotting, outperforming the previous state-of-the-art segmentation-free approaches by a large margin, and in some cases even segmentation-based approaches. One interesting real-life application of our approach is to help historians to find and count specific words in court records that are related to women's sustenance activities and division of labor. We provide promising preliminary experiments that validate our method on this task.