An Empirical Characterization of Event Sourced Systems and Their Schema Evolution -- Lessons from Industry

TL;DR

This paper empirically investigates event sourced systems in industry, revealing key challenges and tactics for schema evolution, based on qualitative data from practitioners and 19 real-world systems.

Contribution

It provides the first detailed empirical characterization of event sourced systems and their schema evolution challenges, grounded in practitioner experience.

Findings

Identifies five major challenges in event sourcing systems.

Proposes five tactics for managing schema evolution.

Provides insights into industry practices and decision factors.

Abstract

Event sourced systems are increasing in popularity because they are reliable, flexible, and scalable. In this article, we point a microscope at a software architecture pattern that is rapidly gaining popularity in industry, but has not received as much attention from the scientific community. We do so through constructivist grounded theory, which proves a suitable qualitative method for extracting architectural knowledge from practitioners. Based on the discussion of 19 event sourced systems we explore the rationale for and the context of the event sourcing pattern. A description of the pattern itself and its relation to other patterns as discussed with practitioners is given. The description itself is grounded in the experience of 25 engineers, making it a reliable source for both new practitioners and scientists. We identify five challenges that practitioners experience: event system evolution, the steep learning curve, lack of available technology, rebuilding projections, and data privacy. For the first challenge of event system evolution, we uncover five tactics and solutions that support practitioners in their design choices when developing evolving event sourced systems: versioned events, weak schema, upcasting, in-place transformation, and copy-and-transform.