Climate Change from Large Language Models

TL;DR

This paper develops an automated, multifaceted evaluation framework to assess large language models' understanding of climate change, revealing their strengths and gaps in climate crisis knowledge.

Contribution

It introduces a hybrid data collection method and comprehensive metrics for evaluating LLMs' climate change knowledge, addressing a gap in systematic assessment.

Findings

LLMs have substantial climate-related knowledge.

There are gaps in the timeliness of LLMs' climate information.

The evaluation framework effectively measures multiple aspects of climate understanding.

Abstract

Climate change poses grave challenges, demanding widespread understanding and low-carbon lifestyle awareness. Large language models (LLMs) offer a powerful tool to address this crisis, yet comprehensive evaluations of their climate-crisis knowledge are lacking. This paper proposes an automated evaluation framework to assess climate-crisis knowledge within LLMs. We adopt a hybrid approach for data acquisition, combining data synthesis and manual collection, to compile a diverse set of questions encompassing various aspects of climate change. Utilizing prompt engineering based on the compiled questions, we evaluate the model's knowledge by analyzing its generated answers. Furthermore, we introduce a comprehensive set of metrics to assess climate-crisis knowledge, encompassing indicators from 10 distinct perspectives. These metrics provide a multifaceted evaluation, enabling a nuanced understanding of the LLMs' climate crisis comprehension. The experimental results demonstrate the efficacy of our proposed method. In our evaluation utilizing diverse high-performing LLMs, we discovered that while LLMs possess considerable climate-related knowledge, there are shortcomings in terms of timeliness, indicating a need for continuous updating and refinement of their climate-related content.