Aggregate Representation Measure for Predictive Model Reusability

TL;DR

This paper introduces the Aggregated Representation Measure (ARM), a predictive index that estimates retraining costs of models under distribution shifts, facilitating cost-effective model reuse and sustainability.

Contribution

The paper presents ARM, a novel measure that predicts retraining costs before actual retraining, aiding in efficient and sustainable model reuse under distribution shifts.

Findings

ARM reasonably predicts retraining costs across noise levels.

ARM enables comparison of model architectures for cost-effectiveness.

ARM supports sustainable AI practices by estimating resource requirements.

Abstract

In this paper, we propose a predictive quantifier to estimate the retraining cost of a trained model in distribution shifts. The proposed Aggregated Representation Measure (ARM) quantifies the change in the model's representation from the old to new data distribution. It provides, before actually retraining the model, a single concise index of resources - epochs, energy, and carbon emissions - required for the retraining. This enables reuse of a model with a much lower cost than training a new model from scratch. The experimental results indicate that ARM reasonably predicts retraining costs for varying noise intensities and enables comparisons among multiple model architectures to determine the most cost-effective and sustainable option.