ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the Portuguese Language

TL;DR

This paper introduces ptt5-v2, a Portuguese-specific T5 model, analyzing how different pretraining configurations affect downstream task performance, and achieves state-of-the-art results on several Portuguese NLP benchmarks.

Contribution

It develops and evaluates a Portuguese T5 model with various pretraining settings, providing insights into their effects and releasing pretrained checkpoints and rerankers.

Findings

Pretraining configuration impacts are subtle compared to baseline.

State-of-the-art results achieved on Portuguese downstream tasks.

Pretrained checkpoints and rerankers are publicly released.

Abstract

Despite advancements in Natural Language Processing (NLP) and the growing availability of pretrained models, the English language remains the primary focus of model development. Continued pretraining on language-specific corpora provides a practical solution for adapting models to other languages. However, the impact of different pretraining settings on downstream tasks remains underexplored. This work introduces $\texttt{ptt5-v2}$, investigating the continued pretraining of T5 models for Portuguese. We first develop a baseline set of settings and pretrain models with sizes up to 3B parameters. Finetuning on three Portuguese downstream tasks (assin2 STS, assin2 RTE, and TweetSentBR) yields SOTA results on the latter two. We then explore the effects of different pretraining configurations, including pretraining data quality, optimization strategies, and multi-epoch pretraining. Perhaps surprisingly, their impact remains subtle compared to our baseline. We release $\texttt{ptt5-v2}$ pretrained checkpoints and their MonoT5-based finetuned $\texttt{MonoPTT5}$ rerankers on HuggingFace in their respective collections at \url{https://huggingface.co/unicamp-dl}.