Tucano 2 Cool: Better Open Source LLMs for Portuguese

TL;DR

Tucano 2 is an open-source suite of Portuguese LLMs with improved datasets, training recipes, and evaluation methods, achieving state-of-the-art performance and fostering community access and reproducibility.

Contribution

We introduce Tucano 2, a comprehensive open-source Portuguese LLM suite with enhanced datasets, training procedures, and evaluation tools, setting new performance standards in the field.

Findings

Achieved state-of-the-art results on Portuguese benchmarks.

Developed new datasets and training recipes for better domain adaptation.

Provided open access to all artifacts for community use.

Abstract

We present Tucano 2, a fully open suite of large language models (LLMs) with 0.5-3.7 billion parameters, designed to address certain gaps in open-source development for Portuguese LLMs. Following our previous works, we now extend our dataset, GigaVerbo-v2, to a new degree of quality and scale, while also introducing a new synthetic dataset, GigaVerbo-v2 Synth, aimed at filling missing gaps in GigaVerbo-v2, and two post-training datasets, GigaVerbo-v2 SFT and GigaVerbo-v2 Preferences, that allow Portuguese LLMs to be trained in domains like retrieval augmented generation, coding, tool use, chain-of-thought reasoning, and many other domains of interest. Through extensive ablation studies, we design both pretraining and continual pretraining recipes for the Tucano 2 suite (Base, Instruct, and Think), which achieve state-of-the-art performance on several Portuguese-language modeling benchmarks. We also extend and refine the evaluation harness introduced in our earlier work, yielding a comprehensive evaluation suite that provides strong signals across different pretraining, continual pretraining, and post-training regimes. All artifacts associated with Tucano 2 are openly released, including training recipes, logs, and source code, ensuring that our work is reproducible, accessible, and extendable by the broader Portuguese NLP community.