On the Adversarial Robustness of 3D Large Vision-Language Models

TL;DR

This paper systematically studies the adversarial robustness of 3D vision-language models, revealing significant vulnerabilities under untargeted attacks and emphasizing the need for robustness improvements in safety-critical applications.

Contribution

It introduces the first systematic analysis of adversarial attacks on 3D VLMs and proposes two novel attack strategies to evaluate their robustness.

Findings

3D VLMs are vulnerable to untargeted adversarial attacks

They show more resilience against targeted attacks compared to 2D models

Highlighting the need for robustness improvements in 3D VLMs

Abstract

3D Vision-Language Models (VLMs), such as PointLLM and GPT4Point, have shown strong reasoning and generalization abilities in 3D understanding tasks. However, their adversarial robustness remains largely unexplored. Prior work in 2D VLMs has shown that the integration of visual inputs significantly increases vulnerability to adversarial attacks, making these models easier to manipulate into generating toxic or misleading outputs. In this paper, we investigate whether incorporating 3D vision similarly compromises the robustness of 3D VLMs. To this end, we present the first systematic study of adversarial robustness in point-based 3D VLMs. We propose two complementary attack strategies: \textit{Vision Attack}, which perturbs the visual token features produced by the 3D encoder and projector to assess the robustness of vision-language alignment; and \textit{Caption Attack}, which directly manipulates output token sequences to evaluate end-to-end system robustness. Each attack includes both untargeted and targeted variants to measure general vulnerability and susceptibility to controlled manipulation. Our experiments reveal that 3D VLMs exhibit significant adversarial vulnerabilities under untargeted attacks, while demonstrating greater resilience against targeted attacks aimed at forcing specific harmful outputs, compared to their 2D counterparts. These findings highlight the importance of improving the adversarial robustness of 3D VLMs, especially as they are deployed in safety-critical applications.