Safe Semantics, Unsafe Interpretations: Tackling Implicit Reasoning Safety in Large Vision-Language Models

TL;DR

This paper identifies a safety vulnerability in large vision-language models caused by implicit reasoning, introduces a dataset to study this issue, and demonstrates that simple in-context learning methods can mitigate these risks.

Contribution

It defines Implicit Reasoning Safety in LVLMs, introduces the SSUI dataset, and shows that in-context learning can reduce implicit reasoning threats.

Findings

SSUI dataset reveals implicit reasoning safety issues

In-context learning mitigates unsafe outputs

Highlights need for improved cross-modal reasoning

Abstract

Large Vision-Language Models face growing safety challenges with multimodal inputs. This paper introduces the concept of Implicit Reasoning Safety, a vulnerability in LVLMs. Benign combined inputs trigger unsafe LVLM outputs due to flawed or hidden reasoning. To showcase this, we developed Safe Semantics, Unsafe Interpretations, the first dataset for this critical issue. Our demonstrations show that even simple In-Context Learning with SSUI significantly mitigates these implicit multimodal threats, underscoring the urgent need to improve cross-modal implicit reasoning.