Diagnosing Moral Reasoning Acquisition in Language Models: Pragmatics and Generalization

TL;DR

This paper investigates whether current learning paradigms can enable large language models to acquire moral reasoning, highlighting a pragmatic dilemma that limits their ability to generalize moral understanding from discourse.

Contribution

The study introduces the concept of the pragmatic dilemma, revealing fundamental limitations of existing learning paradigms in enabling LLMs to generalize moral reasoning.

Findings

Performance improvements are similar to semantic tasks.

Pragmatic nature of morals affects LLMs' reasoning.

Pragmatic dilemma limits moral generalization.

Abstract

Ensuring that Large Language Models (LLMs) return just responses which adhere to societal values is crucial for their broader application. Prior research has shown that LLMs often fail to perform satisfactorily on tasks requiring moral cognizance, such as ethics-based judgments. While current approaches have focused on fine-tuning LLMs with curated datasets to improve their capabilities on such tasks, choosing the optimal learning paradigm to enhance the ethical responses of LLMs remains an open research debate. In this work, we aim to address this fundamental question: can current learning paradigms enable LLMs to acquire sufficient moral reasoning capabilities? Drawing from distributional semantics theory and the pragmatic nature of moral discourse, our analysis indicates that performance improvements follow a mechanism similar to that of semantic-level tasks, and therefore remain affected by the pragmatic nature of morals latent in discourse, a phenomenon we name the pragmatic dilemma. We conclude that this pragmatic dilemma imposes significant limitations on the generalization ability of current learning paradigms, making it the primary bottleneck for moral reasoning acquisition in LLMs.