CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

TL;DR

This paper introduces a new open leaderboard and benchmarks for evaluating large language models specifically for European Portuguese, focusing on language performance, cultural alignment, and safety measures.

Contribution

It presents the first dedicated leaderboard and novel benchmarks for assessing LLMs in European Portuguese, including cultural and safety aspects.

Findings

Established a publicly accessible Portuguese LLM leaderboard

Developed benchmarks for cultural and safety evaluation

Provided insights into LLM performance for European Portuguese

Abstract

This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT), and on its associated benchmarks. This leaderboard comes as a way to address a gap in the evaluation of LLM for European Portuguese, which so far had no leaderboard dedicated to this variant of the language. The paper also reports on novel benchmarks, including some that address aspects of performance that so far have not been available in benchmarks for European Portuguese, namely model safeguards and alignment to Portuguese culture. The leaderboard is available at https://huggingface.co/spaces/PORTULAN/portuguese-llm-leaderboard.