Probabilistic Aggregation and Targeted Embedding Optimization for Collective Moral Reasoning in Large Language Models

TL;DR

This paper introduces a novel framework that combines multiple large language models' moral judgments into a consensus, and fine-tunes individual models through targeted embedding optimization to enhance moral alignment and consistency.

Contribution

It presents a new aggregation and fine-tuning method that improves moral reasoning consistency across models by synthesizing judgments and aligning embeddings to moral theories.

Findings

Enhanced moral judgment consensus among models

Improved individual model fidelity to moral standards

Demonstrated robustness on large-scale social dilemmas

Abstract

Large Language Models (LLMs) have shown impressive moral reasoning abilities. Yet they often diverge when confronted with complex, multi-factor moral dilemmas. To address these discrepancies, we propose a framework that synthesizes multiple LLMs' moral judgments into a collectively formulated moral judgment, realigning models that deviate significantly from this consensus. Our aggregation mechanism fuses continuous moral acceptability scores (beyond binary labels) into a collective probability, weighting contributions by model reliability. For misaligned models, a targeted embedding-optimization procedure fine-tunes token embeddings for moral philosophical theories, minimizing JS divergence to the consensus while preserving semantic integrity. Experiments on a large-scale social moral dilemma dataset show our approach builds robust consensus and improves individual model fidelity. These findings highlight the value of data-driven moral alignment across multiple models and its potential for safer, more consistent AI systems.