NorwAI's Large Language Models: Technical Report

TL;DR

NorwAI developed a family of Norwegian and Scandinavian language models based on Transformer architectures, optimized through extensive training and fine-tuning, to improve NLP performance and accessibility for underrepresented languages.

Contribution

The paper introduces a new suite of Norwegian-focused large language models built on diverse Transformer architectures, with detailed training, fine-tuning, and deployment strategies, and open access for research.

Findings

Models demonstrate strong performance on real-world tasks.

Instruction-tuned variants show effective assistant capabilities.

Open availability supports further research and application.

Abstract

Norwegian, spoken by approximately five million people, remains underrepresented in many of the most significant breakthroughs in Natural Language Processing (NLP). To address this gap, the NorLLM team at NorwAI has developed a family of models specifically tailored to Norwegian and other Scandinavian languages, building on diverse Transformer-based architectures such as GPT, Mistral, Llama2, Mixtral and Magistral. These models are either pretrained from scratch or continually pretrained on 25B - 88.45B tokens, using a Norwegian-extended tokenizer and advanced post-training strategies to optimize performance, enhance robustness, and improve adaptability across various real-world tasks. Notably, instruction-tuned variants (e.g., Mistral-7B-Instruct and Mixtral-8x7B-Instruct) showcase strong assistant-style capabilities, underscoring their potential for practical deployment in interactive and domain-specific applications. The NorwAI large language models are openly available to Nordic organizations, companies and students for both research and experimental use. This report provides detailed documentation of the model architectures, training data, tokenizer design, fine-tuning strategies, deployment, and evaluations.