The Accuracy, Robustness, and Readability of LLM-Generated Sustainability-Related Word Definitions

TL;DR

This study evaluates the quality of climate-related definitions generated by LLMs compared to official standards, focusing on adherence, robustness, and readability to enhance environmental communication.

Contribution

It provides a comparative analysis of LLM-generated climate definitions against official glossaries, highlighting strengths and limitations in adherence and readability.

Findings

LLMs achieved an average adherence of 0.57-0.59

Generated definitions are harder to read than official ones

Models better handle ambiguous or multi-definition terms

Abstract

A common language with standardized definitions is crucial for effective climate discussions. However, concerns exist about LLMs misrepresenting climate terms. We compared 300 official IPCC glossary definitions with those generated by GPT-4o-mini, Llama3.1 8B, and Mistral 7B, analyzing adherence, robustness, and readability using SBERT sentence embeddings. The LLMs scored an average adherence of $0.57-0.59 \pm 0.15$, and their definitions proved harder to read than the originals. Model-generated definitions vary mainly among words with multiple or ambiguous definitions, showing the potential to highlight terms that need standardization. The results show how LLMs could support environmental discourse while emphasizing the need to align model outputs with established terminology for clarity and consistency.