JASMINE: Arabic GPT Models for Few-Shot Learning

TL;DR

JASMINE introduces a suite of large-scale Arabic autoregressive Transformer models and a comprehensive benchmark, addressing the gap in understanding and evaluating GPT models in diverse Arabic linguistic and cultural contexts.

Contribution

The paper presents the first large-scale Arabic GPT models and a dedicated benchmark for evaluation, including social bias and harm assessments.

Findings

JASMINE models perform strongly on NLP tasks in Arabic.

The benchmark reveals social biases and harms in Arabic GPT models.

JASMINE models excel in few-shot learning scenarios.

Abstract

Scholarship on generative pretraining (GPT) remains acutely Anglocentric, leaving serious gaps in our understanding of the whole class of autoregressive models. For example, we have little knowledge about the potential of these models and their societal impacts in diverse linguistic and cultural settings. We alleviate this issue for Arabic, a wide collection of languages and dialectal varieties with more than 400 million population, by introducing JASMINE. JASMINE is a suite of powerful Arabic autoregressive Transformer language models ranging in size between 300 million-6.7 billion parameters pretrained on a large and diverse dataset (~ 235 GB of text). We also carefully design and release a comprehensive benchmark for both automated and human evaluation of Arabic autoregressive models, with coverage of potential social biases, harms, and toxicity. Using our novel benchmark, we evaluate JASMINE extensively showing powerful performance intrinsically as well as in few-shot learning on a wide range of NLP tasks. We aim to responsibly release our models and evaluation benchmark with interested researchers, along with code for experimenting with them.