Common Sense vs. Morality: The Curious Case of Narrative Focus Bias in LLMs

TL;DR

This paper reveals that large language models tend to focus more on moral reasoning than on commonsense understanding, especially when contradictions are linked to secondary characters, highlighting a bias that affects their reasoning robustness.

Contribution

The study introduces CoMoral, a new benchmark dataset for testing LLMs on moral dilemmas with embedded commonsense contradictions, and analyzes the narrative focus bias in existing models.

Findings

LLMs struggle to identify contradictions without prior signals.

Models detect contradictions more easily when linked to secondary characters.

A pervasive narrative focus bias affects LLMs' reasoning capabilities.

Abstract

Large Language Models (LLMs) are increasingly deployed across diverse real-world applications and user communities. As such, it is crucial that these models remain both morally grounded and knowledge-aware. In this work, we uncover a critical limitation of current LLMs -- their tendency to prioritize moral reasoning over commonsense understanding. To investigate this phenomenon, we introduce CoMoral, a novel benchmark dataset containing commonsense contradictions embedded within moral dilemmas. Through extensive evaluation of ten LLMs across different model sizes, we find that existing models consistently struggle to identify such contradictions without prior signal. Furthermore, we observe a pervasive narrative focus bias, wherein LLMs more readily detect commonsense contradictions when they are attributed to a secondary character rather than the primary (narrator) character. Our comprehensive analysis underscores the need for enhanced reasoning-aware training to improve the commonsense robustness of large language models.