MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks

TL;DR

MCIF is a comprehensive, human-annotated benchmark designed to evaluate crosslingual and multimodal instruction-following capabilities of large language models across diverse tasks, modalities, and languages, based on scientific talks.

Contribution

It introduces the first benchmark that jointly assesses crosslingual, multimodal instruction following with human annotations across multiple languages and modalities.

Findings

Universal challenges identified across models and tasks.

Significant room for improvement in multimodal, crosslingual instruction following.

Benchmark covers four macro-tasks and three modalities.

Abstract

Recent advances in large language models have laid the foundation for multimodal LLMs (MLLMs), which unify text, speech, and vision within a single framework. As these models are rapidly evolving toward general-purpose instruction following across diverse and complex tasks, a key frontier is evaluating their crosslingual and multimodal capabilities over both short- and long-form inputs. However, existing benchmarks fall short in evaluating these dimensions jointly: they are often limited to English, mostly focus on a single modality at a time, rely on short-form inputs, or lack human annotations--hindering comprehensive assessment of model performance across languages, modalities, and task complexity. To address these gaps, we introduce MCIF (Multimodal Crosslingual Instruction Following), the first crosslingual human-annotated benchmark based on scientific talks on NLP and beyond. MCIF evaluates instruction following in crosslingual, multimodal settings over different input lengths and spans four macro-tasks: recognition, translation, question answering, and summarization. It covers three core modalities (speech, vision, and text) and four diverse languages (English, German, Italian, and Chinese), fully aligned across all dimensions. This parallel design enables a systematic evaluation of MLLMs' abilities to interpret instructions across languages and effectively integrate multimodal contextual information. Our benchmarking and analysis of 23 models highlight universal challenges across modalities and tasks, indicating substantial room for improvement in future MLLMs development. MCIF is released under CC-BY 4.0 license to promote open research.