Common errors in Generative AI systems used for knowledge extraction in the climate action domain

TL;DR

This paper discusses common errors made by Generative AI systems, especially LLMs like GPT, when used for extracting climate action knowledge, highlighting their limitations and reliability issues.

Contribution

It provides an analysis of the types of errors in LLM outputs in the context of climate knowledge extraction and emphasizes the need for careful use and validation.

Findings

LLMs can generate convincing but unreliable climate knowledge.

Errors include factual inaccuracies and probabilistic hallucinations.

The paper underscores the importance of validation in climate-related AI applications.

Abstract

Large Language Models (LLMs) and, more specifically, the Generative Pre-Trained Transformers (GPT) can help stakeholders in climate action explore digital knowledge bases and extract and utilize climate action knowledge in a sustainable manner. However, LLMs are "probabilistic models of knowledge bases" that excel at generating convincing texts but cannot be entirely relied upon due to the probabilistic nature of the information produced. This brief report illustrates the problem space with examples of LLM responses to some of the questions of relevance to climate action.