BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams
Thales Sales Almeida, Thiago Laitz, Giovana K. Bon\'as, Rodrigo, Nogueira

TL;DR
BLUEX is a new Portuguese-language benchmark dataset based on Brazilian university entrance exams, designed to evaluate and advance NLP models' understanding and reasoning in Portuguese, especially in multimodal contexts.
Contribution
The paper introduces BLUEX, a high-quality, annotated dataset of recent Brazilian university entrance exams, filling a gap in Portuguese NLP evaluation resources and establishing a new benchmark.
Findings
State-of-the-art LMs perform below human level on BLUEX
BLUEX enables evaluation of multimodal reasoning in Portuguese
The dataset is publicly available for further research
Abstract
One common trend in recent studies of language models (LMs) is the use of standardized tests for evaluation. However, despite being the fifth most spoken language worldwide, few such evaluations have been conducted in Portuguese. This is mainly due to the lack of high-quality datasets available to the community for carrying out evaluations in Portuguese. To address this gap, we introduce the Brazilian Leading Universities Entrance eXams (BLUEX), a dataset of entrance exams from the two leading universities in Brazil: UNICAMP and USP. The dataset includes annotated metadata for evaluating the performance of NLP models on a variety of subjects. Furthermore, BLUEX includes a collection of recently administered exams that are unlikely to be included in the training data of many popular LMs as of 2023. The dataset is also annotated to indicate the position of images in each question,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Text Readability and Simplification
