BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams

TL;DR

BLUEX is a new Portuguese-language benchmark dataset based on Brazilian university entrance exams, designed to evaluate and advance NLP models' understanding and reasoning in Portuguese, especially in multimodal contexts.

Contribution

The paper introduces BLUEX, a high-quality, annotated dataset of recent Brazilian university entrance exams, filling a gap in Portuguese NLP evaluation resources and establishing a new benchmark.

Findings

State-of-the-art LMs perform below human level on BLUEX

BLUEX enables evaluation of multimodal reasoning in Portuguese

The dataset is publicly available for further research

Abstract

One common trend in recent studies of language models (LMs) is the use of standardized tests for evaluation. However, despite being the fifth most spoken language worldwide, few such evaluations have been conducted in Portuguese. This is mainly due to the lack of high-quality datasets available to the community for carrying out evaluations in Portuguese. To address this gap, we introduce the Brazilian Leading Universities Entrance eXams (BLUEX), a dataset of entrance exams from the two leading universities in Brazil: UNICAMP and USP. The dataset includes annotated metadata for evaluating the performance of NLP models on a variety of subjects. Furthermore, BLUEX includes a collection of recently administered exams that are unlikely to be included in the training data of many popular LMs as of 2023. The dataset is also annotated to indicate the position of images in each question, providing a valuable resource for advancing the state-of-the-art in multimodal language understanding and reasoning. We describe the creation and characteristics of BLUEX and establish a benchmark through experiments with state-of-the-art LMs, demonstrating its potential for advancing the state-of-the-art in natural language understanding and reasoning in Portuguese. The data and relevant code can be found at https://github.com/Portuguese-Benchmark-Datasets/BLUEX