Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Models

TL;DR

This paper introduces a novel background-consistent object concealment attack on vision-language models that effectively hides objects without causing hallucination, outperforming previous suppression-based methods.

Contribution

The authors propose a new attack method that re-encodes objects to match background semantics, avoiding hallucination and improving concealment effectiveness in vision-language models.

Findings

Reduces hallucination by up to 3 times compared to prior methods.

Preserves up to 86% of non-target objects during concealment.

Effectively conceals objects without creating semantic gaps.

Abstract

Vision-language models (VLMs) have recently shown remarkable capabilities in visual understanding and generation, but remain vulnerable to adversarial manipulations of visual content. Prior object-hiding attacks primarily rely on suppressing or blocking region-specific representations, often creating semantic gaps that inadvertently induce hallucination, where models invent plausible but incorrect objects. In this work, we demonstrate that hallucination arises not from object absence per se, but from semantic discontinuity introduced by such suppression-based attacks. We propose a new class of \emph{background-consistent object concealment} attacks, which hide target objects by re-encoding their visual representations to be statistically and semantically consistent with surrounding background regions. Crucially, our approach preserves token structure and attention flow, avoiding representational voids that trigger hallucination. We present a pixel-level optimization framework that enforces background-consistent re-encoding across multiple transformer layers while preserving global scene semantics. Extensive experiments on state-of-the-art vision-language models show that our method effectively conceals target objects while preserving up to $86\%$ of non-target objects and reducing grounded hallucination by up to $3\times$ compared to attention-suppression-based attacks.