Implicit Language Model in LSTM for OCR

TL;DR

This paper investigates how LSTM neural networks implicitly learn language models in OCR tasks, showing they utilize up to 5 characters of context and improve recognition accuracy by 2.4% CER.

Contribution

It is the first study to characterize the implicit language model learned by LSTMs in OCR, quantifying its context size and impact on performance.

Findings

LSTMs learn an implicit language model in OCR.

The implicit LM uses up to 5 characters of context.

LSTM improves CER by 2.4% on synthetic test set.

Abstract

Neural networks have become the technique of choice for OCR, but many aspects of how and why they deliver superior performance are still unknown. One key difference between current neural network techniques using LSTMs and the previous state-of-the-art HMM systems is that HMM systems have a strong independence assumption. In comparison LSTMs have no explicit constraints on the amount of context that can be considered during decoding. In this paper we show that they learn an implicit LM and attempt to characterize the strength of the LM in terms of equivalent n-gram context. We show that this implicitly learned language model provides a 2.4\% CER improvement on our synthetic test set when compared against a test set of random characters (i.e. not naturally occurring sequences), and that the LSTM learns to use up to 5 characters of context (which is roughly 88 frames in our configuration). We believe that this is the first ever attempt at characterizing the strength of the implicit LM in LSTM based OCR systems.