(Re)Defining Expertise in Machine Learning Development

TL;DR

This paper systematically reviews how expertise is defined and recognized in machine learning research, aiming to create a taxonomy that clarifies roles and highlights opportunities for better engagement of experts.

Contribution

It provides a comprehensive taxonomy of expertise in ML development, addressing the lack of explicit definitions and understanding of expert roles.

Findings

Identifies diverse bases for defining expertise in ML

Highlights gaps in current expert engagement practices

Proposes a taxonomy to improve expert involvement

Abstract

Domain experts are often engaged in the development of machine learning systems in a variety of ways, such as in data collection and evaluation of system performance. At the same time, who counts as an 'expert' and what constitutes 'expertise' is not always explicitly defined. In this project, we conduct a systematic literature review of machine learning research to understand 1) the bases on which expertise is defined and recognized and 2) the roles experts play in ML development. Our goal is to produce a high-level taxonomy to highlight limits and opportunities in how experts are identified and engaged in ML research.