AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese

TL;DR

AMALIA is an open-source large language model specifically trained for European Portuguese, with new benchmarks and datasets to better evaluate and improve pt-PT language capabilities.

Contribution

The paper introduces AMALIA, the first fully open LLM focused on European Portuguese, along with new benchmarks and datasets for native evaluation.

Findings

AMALIA matches strong baselines on translated benchmarks.

AMALIA significantly improves performance on pt-PT-specific tasks.

Targeted training and native benchmarks benefit European Portuguese LLMs.

Abstract

Despite rapid progress in open large language models (LLMs), European Portuguese (pt-PT) remains underrepresented in both training data and native evaluation, with machine-translated benchmarks likely missing the variant's linguistic and cultural nuances. We introduce AMALIA, a fully open LLM that prioritizes pt-PT by using more high-quality pt-PT data during both the mid- and post-training stages. To evaluate pt-PT more faithfully, we release a suite of pt-PT benchmarks that includes translated standard tasks and four new datasets targeting pt-PT generation, linguistic competence, and pt-PT/pt-BR bias. Experiments show that AMALIA matches strong baselines on translated benchmarks while substantially improving performance on pt-PT-specific evaluations, supporting the case for targeted training and native benchmarking for European Portuguese.