Shedding Light on VLN Robustness: A Black-box Framework for Indoor Lighting-based Adversarial Attack

TL;DR

This paper introduces a black-box indoor lighting-based adversarial attack framework that reveals vulnerabilities in vision-and-language navigation agents by manipulating indoor illumination, highlighting real-world robustness issues.

Contribution

The paper proposes ILA, a novel black-box attack method manipulating indoor lighting to evaluate VLN robustness, focusing on realistic lighting variations often overlooked in prior studies.

Findings

ILA significantly increases VLN failure rates.

Lighting manipulations reduce navigation efficiency.

Reveals vulnerabilities to realistic indoor lighting changes.

Abstract

Vision-and-Language Navigation (VLN) agents have made remarkable progress, but their robustness remains insufficiently studied. Existing adversarial evaluations often rely on perturbations that manifest as unusual textures rarely encountered in everyday indoor environments. Errors under such contrived conditions have limited practical relevance, as real-world agents are unlikely to encounter such artificial patterns. In this work, we focus on indoor lighting, an intrinsic yet largely overlooked scene attribute that strongly influences navigation. We propose Indoor Lighting-based Adversarial Attack (ILA), a black-box framework that manipulates global illumination to disrupt VLN agents. Motivated by typical household lighting usage, we design two attack modes: Static Indoor Lighting-based Attack (SILA), where the lighting intensity remains constant throughout an episode, and Dynamic Indoor Lighting-based Attack (DILA), where lights are switched on or off at critical moments to induce abrupt illumination changes. We evaluate ILA on two state-of-the-art VLN models across three navigation tasks. Results show that ILA significantly increases failure rates while reducing trajectory efficiency, revealing previously unrecognized vulnerabilities of VLN agents to realistic indoor lighting variations.