Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages

TL;DR

This paper presents the development of Amharic LLaMA and LLaVA, multimodal large language models tailored for the low-resource Amharic language, utilizing data augmentation and visual instruction tuning to enhance performance.

Contribution

The work introduces the first multimodal LLM for Amharic, combining data augmentation, open source translation, and visual instruction tuning techniques for low-resource languages.

Findings

Achieved improved language understanding in Amharic.

Developed a multimodal model capable of understanding images and text.

Open sourced models and dataset for community use.

Abstract

Large Language Models (LLMs) like GPT-4 and LLaMA have shown incredible proficiency at natural language processing tasks and have even begun to excel at tasks across other modalities such as vision and audio. Despite their success, LLMs often struggle to perform well on low-resource languages because there is so little training data available. This shortcoming is especially prevalent with open source models. In this work, we explore training LLaMA-2 to speak Amharic, a language which is spoken by over 50 million people world wide, but has orders of magnitude less data available than languages like English. We employ methods previously used for training LLMs on other languages with data scarcity, and use open source translation models to perform data augmentation and grow our dataset from millions of tokens to billions. We further enhance the capabilities of our model by connecting an image encoder and training on a translated visual instruction tuning dataset in the same manner as LLaVA, resulting in a multimodal Amharic LLM that can understand images along with text. We introduce an Amharic version of a popular benchmarking dataset to evaluate our work. Our models and dataset are open sourced and available on GitHub.