Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain

TL;DR

This study evaluates the performance and self-evaluation calibration of GPT3.5, Llama2, and Gemma in climate change classification tasks, comparing them to BERT-based models to understand their effectiveness in environmental domains.

Contribution

It provides a comparative analysis of large and small language models' classification accuracy and calibration in climate-related tasks, highlighting their strengths and limitations.

Findings

BERT-based models outperform LLMs and SLM in accuracy

GPT exhibits strong calibration across tasks

Gemma's calibration is inconsistent after initial tasks

Abstract

This paper examines the performance of two Large Language Models (LLMs), GPT3.5 and Llama2 and one Small Language Model (SLM) Gemma, across three different classification tasks within the climate change (CC) and environmental domain. Employing BERT-based models as a baseline, we compare their efficacy against these transformer-based models. Additionally, we assess the models' self-evaluation capabilities by analyzing the calibration of verbalized confidence scores in these text classification tasks. Our findings reveal that while BERT-based models generally outperform both the LLMs and SLM, the performance of the large generative models is still noteworthy. Furthermore, our calibration analysis reveals that although Gemma is well-calibrated in initial tasks, it thereafter produces inconsistent results; Llama is reasonably calibrated, and GPT consistently exhibits strong calibration. Through this research, we aim to contribute to the ongoing discussion on the utility and effectiveness of generative LMs in addressing some of the planet's most urgent issues, highlighting their strengths and limitations in the context of ecology and CC.