EdnaML: A Declarative API and Framework for Reproducible Deep Learning

TL;DR

EdnaML is a framework offering a declarative API for building, managing, and reproducing complex deep learning pipelines, enhancing flexibility and automation in ML workflows.

Contribution

It introduces a layered, declarative API for reproducible deep learning pipelines, combining high-level orchestration with low-level customization.

Findings

Successfully implemented a large-scale fake news classification pipeline.

Demonstrated flexible pipeline management and reproducibility.

Showcased ease of composing ML components with EdnaML.

Abstract

Machine Learning has become the bedrock of recent advances in text, image, video, and audio processing and generation. Most production systems deal with several models during deployment and training, each with a variety of tuned hyperparameters. Furthermore, data collection and processing aspects of ML pipelines are receiving increasing interest due to their importance in creating sustainable high-quality classifiers. We present EdnaML, a framework with a declarative API for reproducible deep learning. EdnaML provides low-level building blocks that can be composed manually, as well as a high-level pipeline orchestration API to automate data collection, data processing, classifier training, classifier deployment, and model monitoring. Our layered API allows users to manage ML pipelines at high-level component abstractions, while providing flexibility to modify any part of it through the building blocks. We present several examples of ML pipelines with EdnaML, including a large-scale fake news labeling and classification system with six sub-pipelines managed by EdnaML.