Beyond the Patch: Exploring Vulnerabilities of Visuomotor Policies via Viewpoint-Consistent 3D Adversarial Object

TL;DR

This paper introduces a viewpoint-consistent 3D adversarial texture method for testing visuomotor policies, revealing vulnerabilities under dynamic viewpoints and demonstrating effectiveness across environments and in real-world scenarios.

Contribution

It proposes a novel differentiable rendering-based adversarial texture optimization for 3D objects, addressing viewpoint variations and enhancing attack transferability.

Findings

Effective under various environmental conditions

Demonstrates black-box transferability

Applicable in real-world robotic scenarios

Abstract

Neural network-based visuomotor policies enable robots to perform manipulation tasks but remain susceptible to perceptual attacks. For example, conventional 2D adversarial patches are effective under fixed-camera setups, where appearance is relatively consistent; however, their efficacy often diminishes under dynamic viewpoints from moving cameras, such as wrist-mounted setups, due to perspective distortions. To proactively investigate potential vulnerabilities beyond 2D patches, this work proposes a viewpoint-consistent adversarial texture optimization method for 3D objects through differentiable rendering. As optimization strategies, we employ Expectation over Transformation (EOT) with a Coarse-to-Fine (C2F) curriculum, exploiting distance-dependent frequency characteristics to induce textures effective across varying camera-object distances. We further integrate saliency-guided perturbations to redirect policy attention and design a targeted loss that persistently drives robots toward adversarial objects. Our comprehensive experiments show that the proposed method is effective under various environmental conditions, while confirming its black-box transferability and real-world applicability.