Declarative Data Services: Structured Agentic Discovery for Composing Data Systems

TL;DR

The paper introduces Declarative Data Services (DDS), a structured framework for agentic discovery in data system composition, effectively handling heterogeneous search spaces and improving convergence over unbounded methods.

Contribution

DDS provides a layered architecture with typed contracts that decompose complex search into manageable sub-searches, enabling effective data system composition from declarative user intent.

Findings

DDS converges on a trading-backend workload where unbounded discovery fails.

Runtime failures are transformed into skill patches for subsequent deployments.

The framework demonstrates practical effectiveness in real-world data-system composition.

Abstract

Agentic discovery has shown that LLM-driven search can find novel algorithms, designs, and code under benchmark conditions. Translating the paradigm to multi-system data backends surfaces a harder problem: the search space is heterogeneous, the verifier is whether a deployed stack actually runs, and composition knowledge is unevenly captured in pretraining. Unbounded agentic discovery, a coding agent iterating on failure-log feedback, fails to converge consistently on a working stack even when iteration and explicit composition knowledge are added. We propose Declarative Data Services (DDS), an architecture for structured agentic discovery of data-system compositions from declarative user intent. The framework owns four typed contracts at successive layers (intent, operator DAG, per-system skills, runtime attribution) that decompose the global search into bounded sub-searches; sub-agents search each typed space, while the framework provides the channels by which knowledge flows forward as inline skill citations and errors route backward as typed signals. As a proof of life on a trading-backend workload, DDS converges where unbounded discovery does not; runtime failures become skill patches that the next deployment cites inline. We position this as an early prototype reporting lessons from real-world data-system composition.