RoQLlama: A Lightweight Romanian Adapted Language Model

TL;DR

This paper introduces RoQLlama, a lightweight Romanian language model based on Llama2, trained with QLoRA, achieving competitive results on Romanian tasks and introducing a new Romanian medical question dataset.

Contribution

It presents RoQLlama-7b, a quantized Romanian LLM that performs well on downstream tasks and introduces the RoMedQA dataset for medical question answering.

Findings

RoQLlama-7b matches or exceeds full-sized model performance.

Higher average scores in few-shot prompts.

Effective Romanian language adaptation with reduced resources.

Abstract

The remarkable achievements obtained by open-source large language models (LLMs) in recent years have predominantly been concentrated on tasks involving the English language. In this paper, we aim to advance the performance of Llama2 models on Romanian tasks. We tackle the problem of reduced computing resources by using QLoRA for training. We release RoQLlama-7b, a quantized LLM, which shows equal or improved results compared to its full-sized counterpart when tested on seven Romanian downstream tasks in the zero-shot setup. Also, it consistently achieves higher average scores across all few-shot prompts. Additionally, we introduce a novel Romanian dataset, namely RoMedQA, which contains single-choice medical questions in Romanian.