Can AI autonomously build, operate, and use the entire data stack?

TL;DR

This paper advocates for a paradigm shift towards fully autonomous AI-managed data estates, emphasizing holistic management of the entire data lifecycle to enhance efficiency and reduce human intervention.

Contribution

It proposes a comprehensive framework for autonomous management of the entire data stack using intelligent agents, moving beyond isolated AI applications in data components.

Findings

AI can potentially automate all stages of the data lifecycle.

Holistic autonomous data management can improve efficiency and reduce manual effort.

Open research questions highlight the need for further development in autonomous data systems.

Abstract

Enterprise data management is a monumental task. It spans data architecture and systems, integration, quality, governance, and continuous improvement. While AI assistants can help specific persona, such as data engineers and stewards, to navigate and configure the data stack, they fall far short of full automation. However, as AI becomes increasingly capable of tackling tasks that have previously resisted automation due to inherent complexities, we believe there is an imminent opportunity to target fully autonomous data estates. Currently, AI is used in different parts of the data stack, but in this paper, we argue for a paradigm shift from the use of AI in independent data component operations towards a more holistic and autonomous handling of the entire data lifecycle. Towards that end, we explore how each stage of the modern data stack can be autonomously managed by intelligent agents to build self-sufficient systems that can be used not only by human end-users, but also by AI itself. We begin by describing the mounting forces and opportunities that demand this paradigm shift, examine how agents can streamline the data lifecycle, and highlight open questions and areas where additional research is needed. We hope this work will inspire lively debate, stimulate further research, motivate collaborative approaches, and facilitate a more autonomous future for data systems.