The Anatomy of a Search and Mining System for Digital Archives

TL;DR

Samtla is a digital humanities system that enables language-agnostic approximate phrase search and document comparison using a character-based n-gram model, supporting textual analysis and pattern discovery in large corpora.

Contribution

The paper introduces Samtla, a novel digital humanities tool employing character-based n-gram models and suffix trees for flexible, language-independent text retrieval and analysis.

Findings

Effective language-agnostic search with high flexibility

Successful case studies demonstrating practical utility

Positive evaluation of ranking performance through crowdsourcing

Abstract

Samtla (Search And Mining Tools with Linguistic Analysis) is a digital humanities system designed in collaboration with historians and linguists to assist them with their research work in quantifying the content of any textual corpora through approximate phrase search and document comparison. The retrieval engine uses a character-based n-gram language model rather than the conventional word-based one so as to achieve great flexibility in language agnostic query processing. The index is implemented as a space-optimised character-based suffix tree with an accompanying database of document content and metadata. A number of text mining tools are integrated into the system to allow researchers to discover textual patterns, perform comparative analysis, and find out what is currently popular in the research community. Herein we describe the system architecture, user interface, models and algorithms, and data storage of the Samtla system. We also present several case studies of its usage in practice together with an evaluation of the systems' ranking performance through crowdsourcing.