The Efficiency Misnomer

TL;DR

This paper critically examines the common assumptions about model efficiency metrics in machine learning, highlighting inconsistencies and proposing better reporting practices to provide a clearer understanding of models' practical costs.

Contribution

It offers a comprehensive analysis of efficiency metrics, discusses their contradictions, and suggests improvements for reporting to enhance transparency and comparability.

Findings

Efficiency metrics often contradict each other.

Incomplete reporting leads to partial conclusions.

Better reporting practices can improve understanding of model costs.

Abstract

Model efficiency is a critical aspect of developing and deploying machine learning models. Inference time and latency directly affect the user experience, and some applications have hard requirements. In addition to inference costs, model training also have direct financial and environmental impacts. Although there are numerous well-established metrics (cost indicators) for measuring model efficiency, researchers and practitioners often assume that these metrics are correlated with each other and report only few of them. In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other. We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models. We further present suggestions to improve reporting of efficiency metrics.