The Effects of Visual Priming on Cooperative Behavior in Vision-Language Models

TL;DR

This study examines how visual priming through images and color cues affects the cooperative decision-making behavior of Vision-Language Models in a simulated game scenario.

Contribution

It reveals the influence of visual inputs on VLM behavior and evaluates mitigation strategies like prompt adjustments and reasoning techniques.

Findings

Visual priming alters VLM decision patterns.

Color cues impact model behavior with varying susceptibility.

Mitigation strategies can reduce priming effects.

Abstract

As Vision-Language Models (VLMs) become increasingly integrated into decision-making systems, it is essential to understand how visual inputs influence their behavior. This paper investigates the effects of visual priming on VLMs' cooperative behavior using the Iterated Prisoner's Dilemma (IPD) as a test scenario. We examine whether exposure to images depicting behavioral concepts (kindness/helpfulness vs. aggressiveness/selfishness) and color-coded reward matrices alters VLM decision patterns. Experiments were conducted across multiple state-of-the-art VLMs. We further explore mitigation strategies including prompt modifications, Chain of Thought (CoT) reasoning, and visual token reduction. Results show that VLM behavior can be influenced by both image content and color cues, with varying susceptibility and mitigation effectiveness across models. These findings not only underscore the importance of robust evaluation frameworks for VLM deployment in visually rich and safety-critical environments, but also highlight how architectural and training differences among models may lead to distinct behavioral responses-an area worthy of further investigation.