Capabilities for Better ML Engineering

TL;DR

This paper proposes a capability-based framework to unify and improve ML engineering practices by specifying model behaviors, with preliminary evidence showing its potential to enhance model generalizability and safety.

Contribution

It introduces a novel capability-based framework for ML engineering that unites various efforts and improves model safety, generalizability, and trustworthiness.

Findings

Capabilities reflect model generalizability

Framework guides ML engineering processes

Potential to build safer, more trustworthy models

Abstract

In spite of machine learning's rapid growth, its engineering support is scattered in many forms, and tends to favor certain engineering stages, stakeholders, and evaluation preferences. We envision a capability-based framework, which uses fine-grained specifications for ML model behaviors to unite existing efforts towards better ML engineering. We use concrete scenarios (model design, debugging, and maintenance) to articulate capabilities' broad applications across various different dimensions, and their impact on building safer, more generalizable and more trustworthy models that reflect human needs. Through preliminary experiments, we show capabilities' potential for reflecting model generalizability, which can provide guidance for ML engineering process. We discuss challenges and opportunities for capabilities' integration into ML engineering.