Open Source Dataset and Machine Learning Techniques for Automatic Recognition of Historical Graffiti

TL;DR

This paper introduces a new dataset of carved historical letters from Kyiv's St. Sophia cathedral and evaluates machine learning models, including CNNs, for automatic recognition of these challenging stone-carved characters.

Contribution

The creation of the CGCL dataset of over 4000 images of carved letters and the assessment of ML models for recognizing difficult stone-carved inscriptions.

Findings

CNN achieved near-perfect AUC of 0.99 on the dataset.

Carved letters are harder to differentiate than handwritten ones.

Open source release of the CGCL dataset for research.

Abstract

Machine learning techniques are presented for automatic recognition of the historical letters (XI-XVIII centuries) carved on the stoned walls of St.Sophia cathedral in Kyiv (Ukraine). A new image dataset of these carved Glagolitic and Cyrillic letters (CGCL) was assembled and pre-processed for recognition and prediction by machine learning methods. The dataset consists of more than 4000 images for 34 types of letters. The explanatory data analysis of CGCL and notMNIST datasets shown that the carved letters can hardly be differentiated by dimensionality reduction methods, for example, by t-distributed stochastic neighbor embedding (tSNE) due to the worse letter representation by stone carving in comparison to hand writing. The multinomial logistic regression (MLR) and a 2D convolutional neural network (CNN) models were applied. The MLR model demonstrated the area under curve (AUC) values for receiver operating characteristic (ROC) are not lower than 0.92 and 0.60 for notMNIST and CGCL, respectively. The CNN model gave AUC values close to 0.99 for both notMNIST and CGCL (despite the much smaller size and quality of CGCL in comparison to notMNIST) under condition of the high lossy data augmentation. CGCL dataset was published to be available for the data science community as an open source resource.