MOROCCO: Model Resource Comparison Framework

TL;DR

MOROCCO is a framework that evaluates NLP models not only on accuracy but also on resource efficiency like memory and inference time, aiding practical model selection.

Contribution

It introduces a comprehensive comparison framework for NLP models that includes resource metrics alongside traditional quality measures.

Findings

Supports over 50 NLU tasks including SuperGLUE

Applicable to multilingual GLUE-like suites

Facilitates resource-aware model evaluation

Abstract

The new generation of pre-trained NLP models push the SOTA to the new limits, but at the cost of computational resources, to the point that their use in real production environments is often prohibitively expensive. We tackle this problem by evaluating not only the standard quality metrics on downstream tasks but also the memory footprint and inference time. We present MOROCCO, a framework to compare language models compatible with \texttt{jiant} environment which supports over 50 NLU tasks, including SuperGLUE benchmark and multiple probing suites. We demonstrate its applicability for two GLUE-like suites in different languages.