Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages

TL;DR

This paper discusses the development of multilingual and multimodal large language models that integrate vision, speech, and text, emphasizing low-resource languages, cost-effective data strategies, and culture-aware evaluation.

Contribution

It provides an overview of recent models, techniques for low-resource data curation, and practical resources for fine-tuning multilingual multimodal models.

Findings

Introduction of recent multilingual multimodal models like PALO and Maya

Methods for low-cost data creation and curation for low-resource languages

Hands-on resources for fine-tuning compact multilingual vision-language models

Abstract

Multimodal LLMs are evolving from vision-language to tri-modality that see, hear, and read, yet pipelines and benchmarks remain English-centric and compute-heavy. The tutorial offers an overview of this emerging research area for multilingual multimodality across text, speech, and vision under limited data/compute budgets, synthesizing foundations, recent multilingual models (PALO, Maya), speech-text LLMs. We cover low-cost data creation/curation; adapter stacks for tri-modal alignment; culture-aware evaluation beyond English and hands on resources for fine-tuning a compact multilingual VLM and wiring a speech->text->LLM pipeline. The content will be delivered as an interactive half-day tutorial, designed for researchers and practitioners working on multilingual, multimodal AI in low-resource language settings.