MARCA: A Checklist-Based Benchmark for Multilingual Web Search

TL;DR

MARCA is a bilingual benchmark for evaluating multilingual web search capabilities of LLMs, focusing on answer completeness and correctness in English and Portuguese.

Contribution

It introduces a new multilingual benchmark with manual questions and rubrics, and evaluates models in different interaction frameworks to analyze performance variability.

Findings

Orchestration improves answer coverage in models.

Large performance differences observed among models.

Significant variability in English to Portuguese transferability.

Abstract

Large language models (LLMs) are increasingly used as sources of information, yet their reliability depends on the ability to search the web, select relevant evidence, and synthesize complete answers. While recent benchmarks evaluate web-browsing and agentic tool use, multilingual settings, and Portuguese in particular, remain underexplored. We present \textsc{MARCA}, a bilingual (English and Portuguese) benchmark for evaluating LLMs on web-based information seeking. \textsc{MARCA} consists of 52 manually authored multi-entity questions, paired with manually validated checklist-style rubrics that explicitly measure answer completeness and correctness. We evaluate 14 models under two interaction settings: a Basic framework with direct web search and scraping, and an Orchestrator framework that enables task decomposition via delegated subagents. To capture stochasticity, each question is executed multiple times and performance is reported with run-level uncertainty. Across models, we observe large performance differences, find that orchestration often improves coverage, and identify substantial variability in how models transfer from English to Portuguese. The benchmark is available at https://github.com/maritaca-ai/MARCA