PePR: Performance Per Resource Unit as a Metric to Promote Small-Scale Deep Learning in Medical Image Analysis

TL;DR

This paper introduces the PePR score, a new metric to evaluate deep learning models based on performance relative to resource consumption, promoting small-scale models for equitable medical image analysis.

Contribution

The paper proposes the PePR score to measure performance per resource unit and demonstrates its usefulness in encouraging resource-efficient models in medical imaging.

Findings

Small-scale models can outperform large models when considering resource efficiency.

Pretrained models fine-tuned on new data reduce computational costs significantly.

Resource-aware evaluation encourages development of more accessible AI solutions.

Abstract

The recent advances in deep learning (DL) have been accelerated by access to large-scale data and compute. These large-scale resources have been used to train progressively larger models which are resource intensive in terms of compute, data, energy, and carbon emissions. These costs are becoming a new type of entry barrier to researchers and practitioners with limited access to resources at such scale, particularly in the Global South. In this work, we take a comprehensive look at the landscape of existing DL models for medical image analysis tasks and demonstrate their usefulness in settings where resources are limited. To account for the resource consumption of DL models, we introduce a novel measure to estimate the performance per resource unit, which we call the PePR score. Using a diverse family of 131 unique DL architectures (spanning 1M to 130M trainable parameters) and three medical image datasets, we capture trends about the performance-resource trade-offs. In applications like medical image analysis, we argue that small-scale, specialized models are better than striving for large-scale models. Furthermore, we show that using existing pretrained models that are fine-tuned on new data can significantly reduce the computational resources and data required compared to training models from scratch. We hope this work will encourage the community to focus on improving AI equity by developing methods and models with smaller resource footprints.