Toward Trustworthy Evaluation of Sustainability Rating Methodologies: A Human-AI Collaborative Framework for Benchmark Dataset Construction

TL;DR

This paper introduces a human-AI collaborative framework for creating trustworthy benchmark datasets to evaluate and improve sustainability rating methodologies, addressing inconsistencies across agencies.

Contribution

It proposes a novel framework combining LLM-guided dataset construction and discrepancy analysis to enhance sustainability ratings' reliability and comparability.

Findings

Framework enables scalable assessment of rating methodologies

Uses LLMs for principled benchmark dataset construction

Provides insights for potential rating adjustments

Abstract

Sustainability or ESG rating agencies use company disclosures and external data to produce scores or ratings that assess the environmental, social, and governance performance of a company. However, sustainability ratings across agencies for a single company vary widely, limiting their comparability, credibility, and relevance to decision-making. To harmonize the rating results, we propose adopting a universal human-AI collaboration framework to generate trustworthy benchmark datasets for evaluating sustainability rating methodologies. The framework comprises two complementary parts: STRIDE (Sustainability Trust Rating & Integrity Data Equation) provides principled criteria and a scoring system that guide the construction of firm-level benchmark datasets using large language models (LLMs), and SR-Delta, a discrepancy-analysis procedural framework that surfaces insights for potential adjustments. The framework enables scalable and comparable assessment of sustainability rating methodologies. We call on the broader AI community to adopt AI-powered approaches to strengthen and advance sustainability rating methodologies that support and enforce urgent sustainability agendas.