How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study

TL;DR

This study evaluates the privacy awareness of vision-language models in realistic physical environments using a new interactive framework, revealing significant perceptual and contextual limitations in current models.

Contribution

Introduces ImmersedPrivacy, an interactive simulation framework for assessing physical-world privacy awareness of VLMs, highlighting their perceptual and contextual shortcomings.

Findings

All models show performance decay with increased scene complexity.

No model exceeds 65% accuracy in social context shifts.

Best model balances task and privacy in only 51% of conflicting commands.

Abstract

As Vision-Language Models (VLMs) are increasingly deployed as autonomous cognitive cores for embodied assistants, evaluating their privacy awareness in physical environments becomes critical. Unlike digital chatbots, these agents operate in intimate spaces, such as homes and hospitals, where they possess the physical agency to observe and manipulate privacy-sensitive information and artifacts. However, current benchmarks remain limited to unimodal, text-based representations that cannot capture the demands of real-world settings. To bridge this gap, we present ImmersedPrivacy, an interactive audio-visual evaluation framework that simulates realistic physical environments using a Unity-based simulator. ImmersedPrivacy evaluates physically grounded privacy awareness across three progressive tiers that test a model's ability to identify sensitive items in cluttered scenes, adapt to shifting social contexts, and resolve conflicts between explicit commands and inferred privacy constraints. Our evaluation of 12 state-of-the-art models reveals consistent deficits. In cluttered scenes, all models exhibit monotonic performance decay as scene complexity grows due to perceptual deficit. When social context shifts, no model exceed 65% selection accuracy. Under conflicting commands, the best model gemini-3.1-pro perfectly balances task completion and privacy preservation in only 51% of cases. These findings reveal that current VLMs in the physical world suffer from perceptual fragility and fail to let their knowledge of privacy cues govern their situated behavior. Our code and data is available at https://github.com/immersed-privacy/immersed-privacy .