Fostering the Ecosystem of Open Neural Encoders for Portuguese with Albertina PT* Family

TL;DR

This paper expands the ecosystem of open-source neural encoders for Portuguese by introducing larger and more efficient models, along with new datasets, to support research and commercial applications in the language.

Contribution

It presents a new 1.5 billion parameter encoder and a 100 million parameter model, extending the Portuguese language model ecosystem with open, high-performance resources.

Findings

Introduction of a 1.5B parameter encoder for Portuguese

Development of a 100M parameter efficient encoder

Creation of new Portuguese datasets based on SuperGLUE

Abstract

To foster the neural encoding of Portuguese, this paper contributes foundation encoder models that represent an expansion of the still very scarce ecosystem of large language models specifically developed for this language that are fully open, in the sense that they are open source and openly distributed for free under an open license for any purpose, thus including research and commercial usages. Like most languages other than English, Portuguese is low-resourced in terms of these foundational language resources, there being the inaugural 900 million parameter Albertina and 335 million Bertimbau. Taking this couple of models as an inaugural set, we present the extension of the ecosystem of state-of-the-art open encoders for Portuguese with a larger, top performance-driven model with 1.5 billion parameters, and a smaller, efficiency-driven model with 100 million parameters. While achieving this primary goal, further results that are relevant for this ecosystem were obtained as well, namely new datasets for Portuguese based on the SuperGLUE benchmark, which we also distribute openly.