VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models

TL;DR

VISOR introduces a novel visual input-based method for controlling vision-language models' behavior, enabling effective, imperceptible output redirection without model access, revealing new security vulnerabilities.

Contribution

It presents VISOR, a universal visual steering technique that achieves bidirectional behavioral control in VLMs using optimized images, compatible with API-based deployments and exposing security risks.

Findings

A single steering image achieves 1-2% performance shift for positive control.

VISOR outperforms system prompting in negative steering by up to 25%.

Maintains 99.9% accuracy on unrelated tasks despite behavioral manipulation.

Abstract

Vision Language Models (VLMs) are increasingly being used in a broad range of applications, bringing their security and behavioral control to the forefront. While existing approaches for behavioral control or output redirection, like system prompting in VLMs, are easily detectable and often ineffective, activation-based steering vectors require invasive runtime access to model internals--incompatible with API-based services and closed-source deployments. We introduce VISOR (Visual Input-based Steering for Output Redirection), a novel method that achieves sophisticated behavioral control through optimized visual inputs alone. By crafting universal steering images that induce target activation patterns, VISOR enables practical deployment across all VLM serving modalities while remaining imperceptible compared to explicit textual instructions. We validate VISOR on LLaVA-1.5-7B across three critical alignment tasks: refusal, sycophancy and survival instinct. A single 150KB steering image matches steering vector performance within 1-2% for positive behavioral shifts while dramatically exceeding it for negative steering--achieving up to 25% shifts from baseline compared to steering vectors' modest changes. Unlike system prompting (3-4% shifts), VISOR provides robust bidirectional control while maintaining 99.9% performance on 14,000 unrelated MMLU tasks. Beyond eliminating runtime overhead and model access requirements, VISOR exposes a critical security vulnerability: adversaries can achieve sophisticated behavioral manipulation through visual channels alone, bypassing text-based defenses. Our work fundamentally re-imagines multimodal model control and highlights the urgent need for defenses against visual steering attacks.