Challenges and Pitfalls of Machine Learning Evaluation and Benchmarking

TL;DR

This paper discusses the challenges and pitfalls in evaluating and benchmarking machine learning models, emphasizing issues with reproducibility, documentation, and comparison across diverse ML artifacts.

Contribution

It provides a comprehensive analysis of common problems in ML evaluation and offers guidelines for better sharing and benchmarking practices.

Findings

Reproducibility issues hinder fair comparison of ML models.

Poor documentation impedes reproducibility and adoption.

Standardized evaluation protocols are needed for reliable benchmarking.

Abstract

An increasingly complex and diverse collection of Machine Learning (ML) models as well as hardware/software stacks, collectively referred to as "ML artifacts", are being proposed - leading to a diverse landscape of ML. These ML innovations proposed have outpaced researchers' ability to analyze, study and adapt them. This is exacerbated by the complicated and sometimes non-reproducible procedures for ML evaluation. A common practice of sharing ML artifacts is through repositories where artifact authors post ad-hoc code and some documentation, but often fail to reveal critical information for others to reproduce their results. This results in users' inability to compare with artifact authors' claims or adapt the model to his/her own use. This paper discusses common challenges and pitfalls of ML evaluation and benchmarking, which can be used as a guideline for ML model authors when sharing ML artifacts, and for system developers when benchmarking or designing ML systems.