Running a Data Integration Lab in the Context of the EHRI Project: Challenges, Lessons Learnt and Future Directions

TL;DR

This paper discusses the experiences, challenges, and lessons learned from running a data integration lab within the EHRI project, aimed at consolidating Holocaust archival sources into a centralized digital platform.

Contribution

It presents a practical model for supporting small institutions in conforming their archival metadata for large-scale data integration efforts.

Findings

Technical challenges in data standardization addressed

Social challenges in engaging small institutions discussed

A replicable framework for data integration proposed

Abstract

Historical study of the Holocaust is commonly hampered by the dispersed and fragmented nature of important archival sources relating to this event. The EHRI project set out to mitigate this problem by building a trans-national network of archives, researchers, and digital practitioners, and one of its main outcomes was the creation of the EHRI Portal, a "virtual observatory" that gathers in one centralised platform descriptions of Holocaust-related archival sources from around the world. In order to build the Portal a strong data identification and integration effort was required, culminating in the project's third phase with the creation of the EHRI-3 data integration lab. The focus of the lab was to lower the bar to participation in the EHRI Portal by providing support to institutions in conforming their archival metadata with that required for integration, ultimately opening the process up to smaller institutions (and even so-called "micro-archives") without the necessary resources to undertake this process themselves. In this paper we present our experiences from running the data integration lab and discuss some of the challenges (both of a technical and social nature), how we tried to overcome them, and the overall lessons learnt. We envisage this work as an archetype upon which other practitioners seeking to pursue similar data integration activities can build their own efforts.