Feature Interactions on Steroids: On the Composition of ML Models

TL;DR

This paper explores the challenges of composing machine learning models without clear specifications, highlighting the importance of feature interactions and proposing insights from software engineering to improve system design, testing, and debugging.

Contribution

It rethinks ML model composition through the lens of feature interactions, drawing parallels with traditional software engineering to address specification issues.

Findings

Weak and wrong specifications are common in practice.

Feature interactions significantly impact ML system behavior.

Insights from software engineering can improve ML system integration.

Abstract

The lack of specifications is a key difference between traditional software engineering and machine learning. We discuss how it drastically impacts how we think about divide-and-conquer approaches to system design, and how it impacts reuse, testing and debugging activities. Traditionally, specifications provide a cornerstone for compositional reasoning and for the divide-and-conquer strategy of how we build large and complex systems from components, but those are hard to come by for machine-learned components. While the lack of specification seems like a fundamental new problem at first sight, in fact software engineers routinely deal with iffy specifications in practice: we face weak specifications, wrong specifications, and unanticipated interactions among components and their specifications. Machine learning may push us further, but the problems are not fundamentally new. Rethinking machine-learning model composition from the perspective of the feature interaction problem, we may even teach us a thing or two on how to move forward, including the importance of integration testing, of requirements engineering, and of design.