Toward a System Building Agenda for Data Integration

TL;DR

This paper advocates for a focused effort on building advanced data integration systems that guide users through workflows, leveraging the Python ecosystem to address current limitations and foster collaborative data management.

Contribution

It proposes a new agenda for developing user-guided data integration systems built on the PyData ecosystem, addressing existing limitations and fostering collaboration.

Findings

DI systems can be significantly improved with workflow guidance

Building on PyData enables flexible, scalable DI tools

Initial implementations show promising results in collaborative settings

Abstract

In this paper we argue that the data management community should devote far more effort to building data integration (DI) systems, in order to truly advance the field. Toward this goal, we make three contributions. First, we draw on our recent industrial experience to discuss the limitations of current DI systems. Second, we propose an agenda to build a new kind of DI systems to address these limitations. These systems guide users through the DI workflow, step by step. They provide tools to address the "pain points" of the steps, and tools are built on top of the Python data science and Big Data ecosystem (PyData). We discuss how to foster an ecosystem of such tools within PyData, then use it to build DI systems for collaborative/cloud/crowd/lay user settings. Finally, we discuss ongoing work at Wisconsin, which suggests that these DI systems are highly promising and building them raises many interesting research challenges.