PhoGPT: Generative Pre-training for Vietnamese
Dat Quoc Nguyen, Linh The Nguyen, Chi Tran, Dung Ngoc Nguyen, Dinh, Phung, Hung Bui

TL;DR
PhoGPT is a new Vietnamese language model series with a 4B-parameter base and chat variant, trained on extensive Vietnamese data, demonstrating superior performance over previous models.
Contribution
Introduction of PhoGPT, a state-of-the-art 4B-parameter Vietnamese language model and its chat variant, trained from scratch and fine-tuned on instructional data.
Findings
PhoGPT outperforms previous open-source Vietnamese models.
The base model is trained on 102B tokens with an 8192 context window.
The chat variant is fine-tuned on 70K prompts and 290K conversations.
Abstract
We open-source a state-of-the-art 4B-parameter generative model series for Vietnamese, which includes the base pre-trained monolingual model PhoGPT-4B and its chat variant, PhoGPT-4B-Chat. The base model, PhoGPT-4B, with exactly 3.7B parameters, is pre-trained from scratch on a Vietnamese corpus of 102B tokens, with an 8192 context length, employing a vocabulary of 20480 token types. The chat variant, PhoGPT-4B-Chat, is the modeling output obtained by fine-tuning PhoGPT-4B on a dataset of 70K instructional prompts and their responses, along with an additional 290K conversations. In addition, we also demonstrate its superior performance compared to previous open-source models. Our PhoGPT models are available at: https://github.com/VinAIResearch/PhoGPT
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Multimodal Machine Learning Applications
MethodsBalanced Selection
