AI4EOSC: a Federated Cloud Platform for Artificial Intelligence in Scientific Research

TL;DR

AI4EOSC is an open-source federated platform that streamlines AI/ML lifecycle management within the European Open Science Cloud, emphasizing FAIR principles, interoperability, and reproducibility across distributed infrastructures.

Contribution

The paper introduces a modular, federated architecture for AI in scientific research, integrating FAIR-by-design principles and provenance tracking within EOSC.

Findings

Successful deployment across heterogeneous cloud providers

Validated scientific use cases demonstrating reduced manual effort

Enhanced reproducibility and interoperability in AI workflows

Abstract

The rapid growth of Artificial Intelligence and Machine Learning in scientific research has highlighted a gap between industry-standard MLOps tools and platforms, and the unique requirements of modern and Open Science, particularly regarding the FAIR (Findable, Accessible, Interoperable, and Reusable) principles. This paper presents AI4EOSC, a federated, open-source platform designed to operationalize the full AI/ML lifecycle within the European Open Science Cloud (EOSC) ecosystem. Our methodology tackles the fragmentation of distributed research infrastructures by integrating a modular and distributed architecture comprising an AI development platform, a serverless AI-as-a-Service layer, and a federated orchestration model that is able to integrate heterogeneous compute and storage resources from distributed e-Infrastructures. AI4EOSC also introduces a ``FAIR-by-design'' approach that enforces metadata standardization (via MLDCAT-AP) and W3C PROV-compliant provenance tracking through a platform-integrated CI/CD pipeline. AI4EOSC added value is demonstrated through the delivery of a diverse set of community installations, showing consistent and seamless deployment across heterogeneous cloud providers. These installations are validated by a set of scientific cases, showing how our work reduces the manual burden on researchers while ensuring high levels of reproducibility and interoperability and providing an unified environment for development, training, and production of AI/ML models in the EOSC.