SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Models

TL;DR

SIF introduces a novel, non-intrusive fingerprinting method for large vision-language models that ensures ownership verification is both stealthy and robust against adversarial attacks and model modifications.

Contribution

The paper proposes SIF, a semantic-preserving fingerprinting framework that transfers text watermarks into visual responses and enhances robustness through worst-case perturbation simulation.

Findings

SIF achieves high stealthiness and robustness in experiments.

It effectively resists semantic divergence attacks and model modifications.

Code is publicly available at the provided GitHub URL.

Abstract

The public accessibility of large vision-language models (LVLMs) raises serious concerns about unauthorized model reuse and intellectual property infringement. Existing ownership verification methods often rely on semantically abnormal queries or out-of-distribution responses as fingerprints, which can be easily detected and removed by adversaries. We expose this vulnerability through a Semantic Divergence Attack (SDA), which identifies and filters fingerprint queries by measuring semantic divergence between a suspect model and a reference model, showing that existing fingerprints are not semantic-preserving and are therefore easy to detect and bypass. To address these limitations, we propose SIF (Semantically In-Distribution Fingerprints), a non-intrusive ownership verification framework that requires no parameter modification. SIF introduces Semantic-Aligned Fingerprint Distillation (SAFD), which transfers text watermarking signals into the visual modality to produce semantically coherent yet fingerprinted responses. In addition, Robust-Fingerprint Optimization (RFO) enhances robustness by simulating worst-case representation perturbations, making the fingerprints resilient to model modifications such as fine-tuning and quantization. Extensive experiments on LLaVA-1.5 and Qwen2.5-VL demonstrate that SIF achieves strong stealthiness and robustness, providing a practical solution for LVLM copyright protection. Code is available at https://github.com/UCF-ML-Research/SIF-VLM-Fingerprint