Enterprise Data Science Platform: A Unified Architecture for Federated Data Access

TL;DR

This paper introduces the Enterprise Data Science Platform (EDSP), a unified architecture based on data lakehouse principles that enables federated data access across multiple analytical environments, reducing data duplication and operational complexity.

Contribution

The paper presents EDSP, a novel multi-layer architecture that facilitates federated data access, interoperability, and reduces data duplication in multi-query engine environments.

Findings

EDSP reduces operational steps by 33-44% compared to traditional data migration approaches.

Major cloud data warehouses and programming environments can directly query EDSP datasets.

End-to-end query response times remain practical, within seconds, despite increased latency.

Abstract

Organizations struggle to share data across departments that have adopted different data analytics platforms. If n datasets must serve m environments, up to n*m replicas can emerge, increasing inconsistency and cost. Traditional warehouses copy data into vendor-specific stores; cross-platform access is hard. This study proposes the Enterprise Data Science Platform (EDSP), which builds on data lakehouse architecture and follows a Write-Once, Read-Anywhere principle. EDSP enables federated data access for multi-query engine environments, targeting data science workloads with periodic data updates and query response times ranging from seconds to minutes. By providing centralized data management with federated access from multiple query engines to the same data sources, EDSP eliminates data duplication and vendor lock-in inherent in traditional data warehouses. The platform employs a four-layer architecture: Data Preparation, Data Store, Access Interface, and Query Engines. This design enforces separation of concerns and reduces the need for data migration when integrating additional analytical environments. Experimental results demonstrate that major cloud data warehouses and programming environments can directly query EDSP-managed datasets. We implemented and deployed EDSP in production, confirming interoperability across multiple query engines. For data sharing across different analytical environments, EDSP achieves a 33-44% reduction in operational steps compared with conventional approaches requiring data migration. Although query latency may increase by up to a factor of 2.6 compared with native tables, end-to-end completion times remain on the order of seconds, maintaining practical performance for analytical use cases. Based on our production experience, EDSP provides practical design guidelines for addressing the data-silo problem in multi-query engine environments.