VinaLLaMA: LLaMA-based Vietnamese Foundation Model

TL;DR

VinaLLaMA is a Vietnamese language model based on LLaMA-2, trained with 800 billion tokens, achieving state-of-the-art performance and cultural understanding for diverse applications.

Contribution

It introduces VinaLLaMA, the first large-scale Vietnamese foundation model with 800 billion tokens and SOTA benchmark results, emphasizing cultural comprehension.

Findings

Achieves SOTA results on VLSP, VMLU, and Vicuna benchmarks.

Demonstrates fluency and cultural understanding in Vietnamese.

Trained on 800 billion tokens, including synthetic data.

Abstract

In this technical report, we present VinaLLaMA, an open-weight, state-of-the-art (SOTA) Large Language Model for the Vietnamese language, built upon LLaMA-2 with an additional 800 billion trained tokens. VinaLLaMA not only demonstrates fluency in Vietnamese but also exhibits a profound understanding of Vietnamese culture, making it a truly indigenous model. VinaLLaMA-7B-chat, trained on 1 million high-quality synthetic samples, achieves SOTA results on key benchmarks, including VLSP, VMLU, and Vicuna Benchmark Vietnamese, marking a significant advancement in the Vietnamese AI landscape and offering a versatile resource for various applications.