Informed AI Regulation: Comparing the Ethical Frameworks of Leading LLM Chatbots Using an Ethics-Based Audit to Assess Moral Reasoning and Normative Values

TL;DR

This study conducts an ethics-based audit of leading LLMs, including GPT-4, evaluating their moral reasoning, normative values, and biases through ethical dilemmas to inform AI safety and regulation.

Contribution

It introduces a systematic, evidence-based ethical audit methodology applied to top LLMs, revealing their moral reasoning capabilities and cultural biases.

Findings

GPT-4 demonstrates a sophisticated ethical framework.

Models show bias towards specific cultural norms.

Many models exhibit authoritarian tendencies.

Abstract

With the rise of individual and collaborative networks of autonomous agents, AI is deployed in more key reasoning and decision-making roles. For this reason, ethics-based audits play a pivotal role in the rapidly growing fields of AI safety and regulation. This paper undertakes an ethics-based audit to probe the 8 leading commercial and open-source Large Language Models including GPT-4. We assess explicability and trustworthiness by a) establishing how well different models engage in moral reasoning and b) comparing normative values underlying models as ethical frameworks. We employ an experimental, evidence-based approach that challenges the models with ethical dilemmas in order to probe human-AI alignment. The ethical scenarios are designed to require a decision in which the particulars of the situation may or may not necessitate deviating from normative ethical principles. A sophisticated ethical framework was consistently elicited in one model, GPT-4. Nonetheless, troubling findings include underlying normative frameworks with clear bias towards particular cultural norms. Many models also exhibit disturbing authoritarian tendencies. Code is available at https://github.com/jonchun/llm-sota-chatbots-ethics-based-audit.