Attacking Motion Estimation with Adversarial Snow

TL;DR

This paper introduces a novel adversarial attack on optical flow estimation using photorealistic, optimized snow, demonstrating significant impact on robustness of existing methods by leveraging realistic weather phenomena.

Contribution

The authors propose a differentiable renderer-based attack that creates realistic snow to fool motion estimation algorithms, a novel approach compared to traditional pixel perturbations.

Findings

Adversarial snow significantly degrades optical flow accuracy.

The attack affects robust methods more than expected.

Realistic weather phenomena can be exploited for adversarial attacks.

Abstract

Current adversarial attacks for motion estimation (optical flow) optimize small per-pixel perturbations, which are unlikely to appear in the real world. In contrast, we exploit a real-world weather phenomenon for a novel attack with adversarially optimized snow. At the core of our attack is a differentiable renderer that consistently integrates photorealistic snowflakes with realistic motion into the 3D scene. Through optimization we obtain adversarial snow that significantly impacts the optical flow while being indistinguishable from ordinary snow. Surprisingly, the impact of our novel attack is largest on methods that previously showed a high robustness to small L_p perturbations.