S2SServiceBench: A Multimodal Benchmark for Last-Mile S2S Climate Services

TL;DR

This paper introduces S2SServiceBench, a comprehensive multimodal benchmark designed to evaluate the ability of large language models and agents to generate reliable, decision-oriented climate services from subseasonal-to-seasonal forecasts across multiple domains.

Contribution

It presents a new benchmark dataset for last-mile climate services, covering diverse applications and levels, and benchmarks current models to identify key challenges in understanding and reasoning under uncertainty.

Findings

Models struggle with understanding service plots and signals.

Operationalizing uncertainty remains a major challenge.

Stable, evidence-based decision analysis is difficult for current models.

Abstract

Subseasonal-to-seasonal (S2S) forecasts play an essential role in providing a decision-critical weeks-to-months planning window for climate resilience and sustainability, yet a growing bottleneck is the last-mile gap: translating scientific forecasts into trusted, actionable climate services, requiring reliable multimodal understanding and decision-facing reasoning under uncertainty. Meanwhile, multimodal large language models (MLLMs) and corresponding agentic paradigms have made rapid progress in supporting various workflows, but it remains unclear whether they can reliably generate decision-making deliverables from operational service products (e.g., actionable signal comprehension, decision-making handoff, and decision analysis & planning) under uncertainty. We introduce S2SServiceBench, a multimodal benchmark for last-mile S2S climate services curated from an operational climate-service system to evaluate this capability. S2SServiceBenchcovers 10 service products with about 150+ expert-selected cases in total, spanning six application domains - Agriculture, Disasters, Energy, Finance, Health, and Shipping. Each case is instantiated at three service levels, yielding around 500 tasks and 1,000+ evaluation items across climate resilience and sustainability applications. Using S2SServiceBench, we benchmark state-of-the-art MLLMs and agents, and analyze performance across products and service levels, revealing persistent challenges in S2S service plot understanding and reasoning - namely, actionable signal comprehension, operationalizing uncertainty into executable handoffs, and stable, evidence-grounded analysis and planning for dynamic hazards-while offering actionable guidance for building future climate-service agents.