Social Norm Reasoning in Multimodal Language Models: An Evaluation

TL;DR

This paper evaluates the ability of multimodal large language models to reason about social norms in text and image scenarios, revealing strengths in text-based reasoning and challenges with complex norms.

Contribution

It introduces an evaluation of five MLLMs' norm reasoning capabilities across multimodal social scenarios, highlighting their comparative performance and limitations.

Findings

MLLMs perform better on text than images.

GPT-4o shows the strongest norm reasoning ability.

Complex norms remain challenging for all models.

Abstract

In Multi-Agent Systems (MAS), agents are designed with social capabilities, allowing them to understand and reason about social concepts such as norms when interacting with others (e.g., inter-robot interactions). In Normative MAS (NorMAS), researchers study how norms develop, and how violations are detected and sanctioned. However, existing research in NorMAS use symbolic approaches (e.g., formal logic) for norm representation and reasoning whose application is limited to simplified environments. In contrast, Multimodal Large Language Models (MLLMs) present promising possibilities to develop software used by robots to identify and reason about norms in a wide variety of complex social situations embodied in text and images. However, prior work on norm reasoning have been limited to text-based scenarios. This paper investigates the norm reasoning competence of five MLLMs by evaluating their ability to answer norm-related questions based on thirty text-based and thirty image-based stories, and comparing their responses against humans. Our results show that MLLMs demonstrate superior performance in norm reasoning in text than in images. GPT-4o performs the best in both modalities offering the most promise for integration with MAS, followed by the free model Qwen-2.5VL. Additionally, all models find reasoning about complex norms challenging.