CLINB: A Climate Intelligence Benchmark for Foundational Models

TL;DR

CLINB introduces a comprehensive benchmark for evaluating large language models' ability to handle complex, multimodal climate change questions, highlighting strengths in knowledge synthesis but challenges in evidence grounding and attribution.

Contribution

This paper presents CLINB, a novel benchmark with real user questions and expert-curated rubrics, to assess LLMs on grounded, multimodal climate knowledge tasks.

Findings

Frontier models show PhD-level understanding and quality in answers.

Models outperform hybrid expert-assisted answers in synthesis.

Significant hallucination rates in references and images.

Abstract

Evaluating how Large Language Models (LLMs) handle complex, specialized knowledge remains a critical challenge. We address this through the lens of climate change by introducing CLINB, a benchmark that assesses models on open-ended, grounded, multimodal question answering tasks with clear requirements for knowledge quality and evidential support. CLINB relies on a dataset of real users' questions and evaluation rubrics curated by leading climate scientists. We implement and validate a model-based evaluation process and evaluate several frontier models. Our findings reveal a critical dichotomy. Frontier models demonstrate remarkable knowledge synthesis capabilities, often exhibiting PhD-level understanding and presentation quality. They outperform "hybrid" answers curated by domain experts assisted by weaker models. However, this performance is countered by failures in grounding. The quality of evidence varies, with substantial hallucination rates for references and images. We argue that bridging this gap between knowledge synthesis and verifiable attribution is essential for the deployment of AI in scientific workflows and that reliable, interpretable benchmarks like CLINB are needed to progress towards building trustworthy AI systems.