Krutrim LLM: Multilingual Foundational Model for over a Billion People

TL;DR

Krutrim LLM is a large multilingual model tailored for India's diverse languages, achieving strong performance across Indic languages and English, and integrated with real-time search to enhance factual accuracy for over a billion users.

Contribution

Introduces Krutrim LLM, a 2 trillion token multilingual model with the largest Indic dataset, addressing data scarcity and linguistic diversity challenges in India.

Findings

Outperforms or matches state-of-the-art models on Indic benchmarks.

Achieves comparable performance to LLAMA-2 on 10 out of 16 tasks.

Balances multilingual fluency with efficient training size.

Abstract

India is a diverse society with unique challenges in developing AI systems, including linguistic diversity, oral traditions, data accessibility, and scalability. Existing foundation models are primarily trained on English, limiting their effectiveness for India's population. Indic languages comprise only 1 percent of Common Crawl corpora despite India representing 18 percent of the global population, leading to linguistic biases. Thousands of regional languages, dialects, and code mixing create additional representation challenges due to sparse training data. We introduce Krutrim LLM, a 2 trillion token multilingual model designed for India's linguistic landscape. It incorporates the largest known Indic dataset, mitigating data scarcity and ensuring balanced performance across dialects. Krutrim outperforms or matches state-of-the-art models on Indic benchmarks while maintaining competitive English performance. Despite being significantly smaller in training flops, Krutrim LLM matches or exceeds models like LLAMA-2 on 10 out of 16 tasks, with an average score of 0.57 versus 0.55. This evidences Krutrim's flexible multilingual fluency across diverse linguistic contexts. Krutrim is integrated with real-time search to improve factual accuracy in conversational AI applications. This enhances accessibility for over 1 billion users worldwide. Through intentional design choices addressing data imbalances, Krutrim LLM signifies meaningful progress in building ethical, globally representative AI models.