A Data Source Dependency Analysis Framework for Large Scale Data Science Projects

TL;DR

This paper introduces an automated framework for mapping data source dependencies in large ML projects, helping MLOps engineers monitor and mitigate data-related issues proactively.

Contribution

It presents a unified static analysis-based system for identifying data source dependencies across various languages, integrated as a REST API for practical deployment.

Findings

Implemented and used by Microsoft MLOps engineers

Reliable identification of data sources across multiple languages

Facilitates proactive data dependency management

Abstract

Dependency hell is a well-known pain point in the development of large software projects and machine learning (ML) code bases are not immune from it. In fact, ML applications suffer from an additional form, namely, "data source dependency hell". This term refers to the central role played by data and its unique quirks that often lead to unexpected failures of ML models which cannot be explained by code changes. In this paper, we present an automated dependency mapping framework that allows MLOps engineers to monitor the whole dependency map of their models in a fast paced engineering environment and thus mitigate ahead of time the consequences of any data source changes (e.g., re-train model, ignore data, set default data etc.). Our system is based on a unified and generic approach, employing techniques from static analysis, from which data sources can be identified reliably for any type of dependency on a wide range of source languages and artefacts. The dependency mapping framework is exposed as a REST web API where the only input is the path to the Git repository hosting the code base. Currently used by MLOps engineers at Microsoft, we expect such dependency map APIs to be adopted more widely by MLOps engineers in the future.