Developing an ESG-Oriented Large Language Model through ESG Practices

TL;DR

This paper introduces an ESG-focused adaptation pipeline for large language models, enhancing their ability to generate environmentally and socially responsible content through domain-aware training strategies.

Contribution

It proposes an ESG-oriented adaptation process using parameter-efficient methods on Qwen-3-4B, improving ESG question answering performance over baseline models.

Findings

ESG-adapted models outperform original and baseline models in ESG question answering.

Parameter-efficient adaptation strategies effectively incorporate ESG principles into LLMs.

The work highlights the importance of responsible, domain-aware language model adaptation.

Abstract

Environmental, Social, and Governance (ESG) considerations play a central role in contemporary financial decision-making. In parallel, Large Language Model (LLM) applications in this domain have primarily emphasized well-defined discriminative tasks, such as classification or scoring, which have proven effective for structured analysis and benchmarking. However, this prevailing focus offers limited support for more interactive and generative ESG scenarios, where embedded domain knowledge and contextual understanding are essential. In this work, we propose an ESG-oriented adaptation pipeline for LLMs that integrates ESG principles not only as a target domain, but also as guiding constraints throughout training and evaluation. Building on the Qwen-3-4B architecture, we explore parameter-efficient adaptation strategies using Low-Rank Adaptation (LoRA) and the Instruction-Residual Method (IRM) to produce three ESG-specialized models. We evaluate the proposed models on ESG question answering under both zero-shot and knowledge-augmented settings, using a diverse set of generative, semantic, readability, and environmental impact metrics. Our results show that the ESG-adapted models consistently outperform their original counterparts and competitive baselines such as Llama-3 and Gemma-3. Although limitations remain in tool-based knowledge integration, this work establishes a foundation for ESG-oriented language generation and highlights the importance of responsible, domain-aware LLM adaptation.